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Abstract 

We prove that when all hyperelliptic curves of genus n > 1 having a rational Weierstrass point 
are ordered by height, the average size of the 2-Selmer group of their Jacobians is equal to 3. 
It follows that (the limsup of) the average rank of the Mordell-Weil group of their Jacobians is 
at most 3/2. 

The method of Chabauty can then be used to obtain an effective bound on the number of rational 
points on most of these hyperelliptic curves; for example, we show that a majority of hyperelliptic 
curves of genus n > 3 with a rational Weierstrass point have fewer than 20 rational points. 

Contents 



1 Introduction 2 

2 The method of proof 4 

3 A representation of the orthogonal group 7 

; 4 Orbits with a fixed characteristic polynomial 9 

4.1 The distinguished orbit 10 

4.2 The remaining orbits via Galois cohomology 13 

5 Hyperelliptic curves 14 

5.1 The 2-torsion subgroup 15 

5.2 A subgroup in the kernel of ?7 15 

6 Orbits over arithmetic fields 17 

6.1 Orbits over a finite field 17 

6.2 Orbits over a non-archimedean local field 18 

6.3 Orbits over C and R 19 

6.4 Orbits over a global field 20 

7 Nilpotent orbits and Vinberg's theory 20 



1 



8 Integral orbits 22 

8.1 Orbits over Z p for p ^ 2 22 

8.2 General integral orbits 25 

9 Construction of fundamental domains 27 

9. 1 Fundamental sets for the action of G(R) on V(R) 28 

9.2 Fundamental sets for the action of G(Z) on G(M) 29 

9.3 Fundamental sets for the action of G(Z) on V(R) 29 

10 Counting irreducible integral points of bounded height 30 

10.1 Averaging over fundamental domains 30 

10.2 An estimate from the geometry of numbers 32 

10.3 Estimates on reducibility 32 

10.4 The main term 36 

10.5 Computation of the volume 38 

10.6 Congruence conditions 39 

10.7 Proof of Propositions 30 and 31 40 

11 Sieving to Selmer elements 41 

11.1 A weighted set U(F) in V(Z) corresponding to a large family F 41 

1 1.2 Local densities of the weighted sets U (F) 42 

11.3 A key uniformity estimate 44 

11.4 Weighted count of elements in U(F) having bounded height 44 

12 Proof of Theorem 1 45 

12.1 The number of hyperelliptic curves of bounded height 45 

12.2 Evaluation of the average size of the 2-Selmer group 45 



1 Introduction 

A hyperelliptic curve C of genus n > 1 over Q with a marked rational Weierstrass point O has an 
affine equation of the form 

= x 2n+l + C2X 2n-l + + + ^ = f ^ (1) 

with rational coefficients c m . The point O lies above x = oo, the polynomial f(x) is separable, and 
the ring of functions which are regular outside of O is the Dedekind domain Q[x, y] = Q[x, ^/ f(x)]. 
The change of variable (x 1 = u 2 x,y' = u 2n+1 y) results in a change in the coefficients c' m = u 2m c m . 
Hence we may assume the coefficients are all integers. These integers are unique if we assume further 
that, for every prime p, the integral coefficients c m are not all divisible by p 2m . In this case we say the 
coefficients are indivisible. 

To make the next two definitions, we assume that C is given by its unique equation with indivis- 
ible integral coefficients. The discriminant of f(x) is a polynomial D(c2, C3, . . . , C2 n +i) of weighted 
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homogeneous degree 2n{2n + 1) in the coefficients c m , where c m has degree m. Since f(x) is sep- 
arable, this discriminant is non-zero. We define the discriminant A of the curve C by the formula 
A(C) := 4 2n D(c2, C3, . . . , C2n+i), and the (naive) height H of the curve C by 

H{C) := mzx{\c k \ 2ni2n+1)/k }l n = + 2 1 . 

We include the expression 2n{2n + 1) in the definition so that the weighted homogeneous degree of 
the height function H is the same as that of the discriminant A. If A(C) is prime to p, the hyperelliptic 
curve C has good reduction modulo p. The height H(C) gives a concrete way to enumerate all of the 
hyperelliptic curves of a fixed genus with a rational Weierstrass point: for any real number X > there 
are clearly only finitely many curves with H(C) < X. 

The discriminant and height extend the classical notions in the case of elliptic curves (which is the 
case n = 1). Any elliptic curve E over Q is given by a unique equation of the form y 2 = x 3 + c 2 x + c 3 , 
where c 2 , c 3 G Z and for all primes p: p 6 \ c 3 whenever p 4 | c 2 . The discriminant is then defined by the 
formula A(E) := 2 4 (-4c 2 i - 27cj) and the naive height by H(E) := max{|c 2 | 3 , |c 3 | 2 }. 

Recall that the 2-Selmer group S 2 (J) of the Jacobian J = Jac(C) of C is a finite subgroup of 
the Galois cohomology group iJ^Q, J [2]), which is defined by local conditions and fits into an exact 
sequence 

-> J(Q)/2J(Q) -> 5 2 (J) -> nij[2] -> 0, 

where III j denotes the Tate-Shafarevich group of J over Q. 
The purpose of this paper is to prove the following theorem. 

Theorem 1 When all hyperelliptic curves of fixed genus n > 1 over Q having a rational Weierstrass 
point are ordered by height, the average size of the 2-Selmer groups of their Jacobians is equal to 3. 

More precisely, we show that 

E//(c)<x#' S 2(Jac(C)) 
lim = 3. 

X ^°° 2^H(C)<X 1 

In fact, we prove that the same result remains true even when we average over any subset of hyperel- 
liptic curves C defined by a finite set of congruence conditions on the coefficients c 2 , c 3 , . . . , c 2n +i- 

Since the 2-rank r 2 (S , 2 ( J)) of the 2-Selmer group of the Jacobian J bounds the rank of the Mordell- 
Weil group J(Q), and since 2r 2 (S' 2 (J)) < 2 T ' 2 ( 52 ( J ^ = #S 2 (J), by taking averages we immediately 
obtain: 

Corollary 2 When all rational hyperelliptic curves of fixed genus n > 1 with a rational Weierstrass 
point are ordered by height, the average 2-rank of the 2-Selmer groups of their Jacobians is at most 3/2. 
Thus the average rank of the Mordell-Weil groups of their Jacobians is at most 3/2. 

Another corollary of Theorem 1 is that the average 2-rank of the 2-torsion subgroup of the Tate- 
Shafarevich group III j [2] is also at most 3/2. The analogous statements for elliptic curves were proven 
in [4]. 

We suspect that the average rank of the Mordell-Weil group is equal to 1/2. However, the fact that 
the average rank is at most 3/2 (independent of the genus) already gives some interesting arithmetic 
applications. Recall that the method of Chabauty [16], as refined by Coleman [17], yields a finite and 
effective bound on the number of rational points on a curve over Q whenever its genus is greater than 
the rank of its Jacobian. Theorem 1 implies 
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Corollary 3 Let 5 n denote the lower density of hyperelliptic curves of genus n with a rational Weier- 
strass point satisfying Chabauty's condition. Then 5 n — > 1 as n -> oo. 

Indeed, we have 5 n > 1 — 2/(2™ — 1), so 5 n — > 1 quite rapidly as n gets large. Thus for an asymptotic 
density of 1 of hyperelliptic curves with a rational Weierstrass point, one can effectively bound the 
number of rational points. 

As an explicit consequence, we may use Theorem 1, together with the arguments in [49], to prove 
the following 

Corollary 4 . 

(a) For any n > 2, a positive proportion of hyperelliptic curves of genus n with a rational Weier- 
strass point have at most 3 rational points. 

(b) For any n > 3, a majority (i.e., a proportion of > 50%) of all hyperelliptic curves of genus n 
with a rational Weierstrass point have fewer than 20 rational points. 

In fact, we will give a lower bound on the proportion in (a) which is independent of the genus n. In (b), 
when n = 2, we may still deduce that more than a quarter of all such curves have fewer than 20 rational 
points. 

The numbers in Corollary 4 can certainly be improved with a more careful analysis. Bjorn Poonen 
and Michael Stoll have recently shown us arguments which imply (provided n > 3) that a positive 
proportion of such curves have at most one rational point, and a majority of such curves have at most 
7 rational points; see [40] for details and further such results. 

Note that Corollary 4(a) is also true when n = 1: it follows from [5] that a positive proportion of 
elliptic curves have only one rational point (the origin of the group law). Meanwhile, Corollary 4(b) is 
not expected to be true when n — 1, since half of all elliptic curves are expected to have rank one and 
thus infinitely many rational points. We suspect that 100% of all hyperelliptic curves of genus n > 2 
over Q with a rational Weierstrass point have only one rational point. 

2 The method of proof 

The proof of Theorem 1 follows the argument in [4] for elliptic curves. There the main idea was to use 
the classical parametrization [7, Lemma 2] of 2-Selmer elements of elliptic curves by certain binary 
quartic forms to transform the problem into one of counting the integral orbits of the group PGL 2 on 
the space Sym 4 of binary quartics. Over Q, the group PGL 2 is isomorphic to the special orthogonal 
group of the three-dimensional quadratic space W of 2 x 2 matrices of trace zero, with quadratic form 
given by the determinant, and the representation Sym 4 of PGL 2 on the space of binary quartic forms 
is given by the action of SO(W) by conjugation on the space V of self-adjoint operators T : W — >• W 
with trace zero [3]. 
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Let W now be a bilinear space of rank 2n + 1 over Q, where the Gram matrix A of the bilinear 
form (w, u) on Q 2n+1 consists of l's on the anti-diagonal and O's elsewhere: 



A 



1 

1 



1 

1 



(2) 



Let SO(W) be the special orthogonal group of W over Q, i.e., the subgroup of Gh(W) defined by the 
algebraic equations (gw,gu) = (w,u) and det(g) = +1. (One can define this group scheme over Z, 
using the unimodular lattice defined by A, but it is only smooth and reductive over Z[l/2]). Let V 
be the representation of SO(W) given by conjugation on the self-adjoint operators T : W — > W of 
trace zero. With respect to the above basis, an operator T is self-adjoint if and only if its matrix M is 
symmetric when reflected about the anti-diagonal. The matrix B = AM is then symmetric in the usual 
sense. The symmetric matrix B is the Gram matrix of the associated bilinear form (w, u)t = (w, Tu), 
and has anti-trace zero. 

The coefficients of the monic characteristic polynomial of T, defined by 

f(x) = det(xl — T) = (-l) n det(xA - B) = x 2n+1 + c 2 x 2n ~ l + ■■■ + c 2n+1 , (3) 

give 2n independent invariant polynomials c 2 , C3, . . . c 2n +i on V, having degrees 2, 3, . . . , 2n + 1 re- 
spectively, which generate the (free) ring of SO (W) -invariants [3]. When the discriminant disc(/) of 
/ is nonzero, we show how classes in the 2-Selmer group of the Jacobian of the hyperelliptic curve 
C with equation y 2 = f(x) over Q correspond to certain orbits of SO(VT)(Q) on V(Q) with these 
polynomial invariants. 

More precisely, to each self-adjoint operator T with invariants c 2 , c 3 , . . . , c 2n+ i satisfying A(T) : = 
4 2n ■ disc(/) 7^ 0, we associate a non-degenerate pencil of quadrics in projective space F(W © Q) = 
p2n+i_ rp WQ q Ua( jrics generating this pencil are Q(w, z) = (w, w) and Q'(w, z) = (w, Tw) + z 2 . The 
discriminant locus disc(xQ-x'Q') of this pencil is a homogeneous polynomial g (x, x') of degree 2n+2 
satisfying #(1,0) = and g(x, 1) = f(x). The Fano variety Ft of maximal linear isotropic subspaces 
of the base locus is smooth of dimension n over Q and forms a principal homogeneous space for the 
Jacobian J of the curve C (see [21]). 

Both the pencil and the Fano variety have an involution r induced by the involution t(w,z) = 
(w, —z) of W © Q. The involution r of F T has 2 2n fixed points over an algebraic closure of Q, which 
form a principal homogeneous space Pt for the 2-torsion subgroup J[2] of the Jacobian J. Indeed, one 
can define the structure of a commutative algebraic group on the disconnected variety G — J Li Ft such 
that the involution — \q of G induces the involution r on the non-trivial component Ft (see [51]). We 
prove that the isomorphism class of the principal homogeneous space P T over Q determines the orbit 
of T. This gives an injective map from the set of rational orbits of SO(W^) on V with characteristic 
polynomial f(x) to the set of elements in the Galois cohomology group // 1 (Q, J[2}), which classifies 
principal homogeneous spaces for this finite group scheme. 

Theorem 5 Let C be the hyperelliptic curve of genus n which is defined by the ajfine equation y 2 = 
f(x), where f(x) is a monic, separable polynomial of degree 2n + 1 over Q. Then the classes in the 
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2-Selmer group of the Jacobian J of C over Q correspond bijectively to the orbits ofSO(W)(Q) on 
self-adjoint operators T : W W with characteristic polynomial f(x) such that the associated Fano 
variety Ft has points over Q v for all places v. 

We call such special orbits of SO(W)(Q) on V(Q) locally soluble. Since they are in bijection with 
elements in the 2-Selmer group of Jac(C), they are finite in number (whereas the number of rational 
orbits with characteristic polynomial f(x) turns out to be infinite). Local solubility is a subtle concept. 
For example, for any d in Q* there is an obvious bijection (T — > dT) between the rational orbits 
with characteristic polynomial f(x) and the rational orbits whose characteristic polynomial f*(x) has 
coefficients c* k = d k Ck- But this bijection does not necessarily preserve the locally soluble orbits, as the 
hyperelliptic curves C and C* are isomorphic over Q only when d is a square. 

When n — 1, the Fano variety F = F T is the intersection of two quadrics in P 3 . This intersection 
defines a curve of genus 1 whose Jacobian is the elliptic curve J = C. The quotient of F by the 
involution r is a curve X of genus 0, and the covering F — > X is ramified at the 4 fixed points 
comprising Pt. Since the curve X injects into the variety of isotropic lines in the split quadratic space 
W of dimension 2n + 1 = 3, we conclude that X is isomorphic to P 1 over Q. (For n > 1, the quotient 
X of F by r embeds as singular subvariety of the Lagrangian Grassmannian of W , with 2 2n double 
points, which in turn embeds in projective space of dimension 2 n — 1 via the spin representation. The 
composition is the Kummer embedding of X.) In the case where n — 1, once we fix an isomorphism 
of X with P 1 , the Fano variety F has an equation of the form z 2 = T(x, y), where T(x, y) is a binary 
quartic form. The orbit is locally soluble if and only if the quartic form T(x, y) represents a square in 
Q v for all places v, and the homogeneous space Pt is trivial if and only if T(x, y) has a linear factor 
over Q. This is the point of view taken in [4], where the latter orbits are called reducible. 

In order to obtain Theorem 1 from Theorem 5, we are reduced to counting the Q-orbits in the rep- 
resentation V that are both locally soluble and have bounded integral invariants. To count these locally 
soluble orbits, we prove that every such orbit of V with integral polynomial invariants has an integral 
representative having those same invariants (at least away from the prime 2). By a suitable adaptation 
of the counting techniques of [1] and [4], we first carry out a count of the total number of integral 
orbits in these representations having bounded height satisfying certain irreducibility conditions over 
Q (to be defined). The primary obstacle in this counting, as in representations encountered previously, 
is that the fundamental region in which one has to count points is not compact but instead has a rather 
complex system of cusps going off to infinity. A priori, it could be difficult to obtain exact counts of 
points of bounded height in the cusps of these fundamental regions. We show however that, for all 
n, most of the integer points in the cusps correspond to points that are reducible; meanwhile, most of 
the points in the main bodies of these fundamental regions are irreducible. The orbits which contain a 
reducible point turn out to correspond to the identity classes in the corresponding 2-Selmer groups. 

Since not all integral orbits correspond to Selmer elements, the proof of Theorem 1 requires a sieve 
to those points that correspond to locally soluble Q-orbits on V, for which the squarefree sieve of [2] is 
applied. By carrying out this sieve, we prove that the average occurring in Theorem 1 arises naturally 
as the sum of two contributions. One comes from the main body of the fundamental region, which 
corresponds to the average number of non-identity elements in the 2-Selmer group and which we show 
is given by the Tamagawa number (= 2) of the adjoint group SO(W) over Q. The other comes from 
the cusp of the fundamental region, which counts the average number (= 1) of identity elements in the 
2-Selmer group. The sum 2+1 = 3 then gives us the average size of the 2-Selmer group, as stated in 
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Theorem 1. 

To obtain Corollary 3, we note that by Theorem 1 the density S n must satisfy 

5 n ■ 1 + (1 - 5 n ) ■ 2 n < 3 (4) 

yielding 5 n > 1 - 2/(2 n - 1). 

To obtain explicit bounds on the number of rational points on these curves, we follow the arguments 
in [49]. Assume that n > 2. Using congruence conditions at the prime p = 3, we may assume that all 
of the hyperelliptic curves C of genus n in our sample have good reduction modulo 3, and that the only 
point on C over Z/3Z is the Weierstrass point at oo. This simply means that the discriminant A of the 
integral polynomial f(x) is prime to 3 and that the three values /(0), f(l), and /(— 1) are all = —1 
modulo 3. Hence any rational point P on C lies in the 3-adic disc reducing to the Weierstrass point. 

Since the average rank of the Jacobians in our sample is 3/2, a positive proportion (indeed, at least 
a third) of these Jacobians have rank zero or one. Assume that this is the case for such a Jacobian J, 
let T be the (finite) torsion subgroup of J(Qs), and let A be the Z 3 -submodule of J(Q 3 )/T which is 
generated by J(Q). Then the Z 3 -rank of A is either zero or one. Let be the Q3-vector space of 
invariant differentials on J, or equivalently the regular 1 -forms on the curve C over Q 3 . The logarithm 
on the formal group gives a Z 3 -bilinear pairing f2 x J(Q 3 )/T — > Q 3 with trivial left and right kernels. 
Let f2 be the subspace of f2 which annihilates the submodule A. The codimension of Q is either zero 
or one, and any differential u in the subspace Q pairs trivially with the class of the divisor (P) — (O), 
for any rational point P. Since P and O both lie in the disc reducing to the Weierstrass point, this 
pairing is computed by 3-adic integration. Namely, the function z = x n /y is a uniformizing parameter 
on this disc which vanishes at O and is taken to its negative by the hyperelliptic involution. If z(P) is 
the parameter of the rational point P and we expand cu = (a + a 2 z 2 + a A z A + ■ ■ ■ )dz, then the integral 
of ijj from O to P is given by the value of the formal integral F(z) = a z + (a 2 /3)z 3 + (a i /5)z 5 + ■ ■ ■ 
at the point z(P) in the maximal ideal of Z 3 . We can choose u integral, and since we are in a subspace 
flo of codimension at most one, the reduction of u has at most a double zero at the Weierstrass point. 
Hence the coefficients a 2 i are all integral, and either a or a 2 is a unit. It follows that the formal integral 
F(z) has at most three zeroes in the disc 3Z 3 , and hence that there are at most three rational points on 
the curve C, proving Corollary 4(a). 

To obtain Corollary 4(b), we note that a density of 1 — 1/7 of hyperelliptic curves with a rational 
Weierstrass point have good reduction at 7. The congruence version of Theorem 1 implies that among 
these curves, the density of curves having rank < 2 is at least 1 — 2/(2 3 — 1). Since #C(F 7 ) < 15, 
Stall's refinement [49, Cor. 6.7] of Coleman's effective bound for Chabauty's method then results in 
#C(Q) < 15+2-2 = 19 for such curves. Hence the majority (indeed greater than (1 — 1)(1 — ^tzy) = 
30/49 ~ 61%) of our curves have fewer than 20 rational points, as desired. 

3 A representation of the orthogonal group 

Let W be a lattice of rank 2n + 1 having a non-degenerate integral bilinear form with signature (n + 
1, n). By this we mean that W is a free abelian group of rank 2n + 1 with a non-degenerate symmetric 
bilinear pairing ( , ) : W x W — > Z having signature (n + 1, n) over E. It follows that the determinant 
of W is equal to (— l) n . 
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Such a lattice is unique up to isomorphism [45, Ch V]. It therefore has an ordered basis 



{e 1 ,e 2 , . . . ,e n ,u,f n , . . . , f 2 , fi} 

with inner products given by 

( e i> e i) = (fi,fj) = ( e U U ) = (fil U ) = 

ipii fj) 

(u, u) = 1. 

The Gram matrix A of the bilinear form with respect to this basis (which we will call the standard 
basis) is the anti-diagonal matrix (2). 

The lattice W gives a split orthogonal space W <E> k over any field k with char(A;) ^ 2. This space 
has dimension 2n + 1 and determinant = (—1)™ in k* /k* 2 . Such an orthogonal space over k is unique 
up to isomorphism, and has disc(W) = (—1)™ det(W) = 1 (see [35]). 

Let T : W — > W be a Z-linear transformation (i.e., an endomorphism of the abelian group W). We 
define the adjoint transformation T* uniquely by the formula 

(Tv, w) = (v, T*w). 

The matrix M of T* with respect to our standard basis is obtained from the matrix of T by reflection 
around the anti-diagonal. In particular, we have the identity det(T) = det(T*). We say that T is 
self-adjoint if T = T*. Any self-adjoint transformation T defines a new symmetric bilinear form on V 
by the formula 

(v, w)t = (v, Tw). 

This bilinear form has symmetric Gram matrix B = AM in our standard basis. 

We say that a Z-linear transformation g : W — > W is orthogonal if it preserves the bilinear 
structure on W, i.e., (gv, gw) = (v, w) for all w E W. In that case g is invertible, with g^ 1 = g*, and 
det(g) — ±1. We define the group scheme G = SO(W) over Z by 

G := SO(W) = {gE GL(W) : g*g = 1, det(g) = 1}. 

Over Z[l/2] the group scheme G is smooth and reductive. It defines the split reductive adjoint 
group with root system of type B n (see [20]). In particular, for every field k with char(/c) ^ 2, the 
group G(k) gives the points of the split special orthogonal group S0 2n +i of the space W Cg> k. 

We note that the group G is not smooth over Z 2 . Indeed, the involution g E G(Z/2Z) with matrix 



(5) 



does not lift to an element of G(Z 2 ), or even to an element of G(Z/4Z). The theory of Bruhat and 
Tits provides a smooth model Q over Z 2 with the same points in etale Z 2 -algebras (see [13]), namely, 
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Q is the normalizer of a parahoric subgroup P of G(Q 2 ). When n = 1, the subgroup P is an Iwahori 
subgroup, and when n > 2, the subgroup P is a maximal parahoric subgroup. The reductive quotient 
of P is the split group S0 2n . We will not need this smooth model in what follows. 

If g is orthogonal and T is self-adjoint, then the transformation gTg^ 1 = gTg* is also self-adjoint. 
The self-adjoint operators form a free submodule Y of End(W), of rank (2n + l)(n + 1), and the 
action of special orthogonal transformations by conjugation T — > gTg^ 1 gives a linear representation 
G — > GL(Y). Over Z[l/2] the representation F is orthogonal, via the non-degenerate symmetric 
pairing (T 1 ,T 2 ) = Trace(TiT 2 ). If we identify the module of self-adjoint operators with the module 
of symmetric matrices, via the map B = AM described above, then Y is isomorphic to the symmetric 
square of the standard representation W over Z[l/2]. 

The trace of an operator gives a non-zero, G-invariant linear form Y — > Z. We define the represen- 
tation V of G over Z as its kernel: 

V = {T : W -> W : T = T*, Trace(T) = 0}. 

The free Z-module V(Z) has rank 2n 2 +3?i, and can be identified with the submodule of symmetric ma- 
trices having anti-trace zero (where the anti-trace is the sum of the matrix entries on the anti-diagonal). 
We are going to study the orbits of G(Z) on V(Z). 

We note that the representation V of G is algebraically irreducible in all characteristics which do 
not divide 2 or 2n + 1. If the characteristic divides 2n + 1, then V contains a copy of the trivial 
representation (as the scalar multiples of the identity operator 7 are self-adjoint and have trace zero) 
and the quotient module is algebraically irreducible. In characteristic 2, there are even more irreducible 
factors in the Jordan-Holder series of V. 

4 Orbits with a fixed characteristic polynomial 

Let k be a field, with char(A;) ^ 2. In this section and the next, we consider the orbits of the group 
G(k) = SO(W)(k) on the representation V(k) = V ® k. A discussion of these orbits can also be 
found in our expository paper [3]. 

Since the group G(k) acts on the operators T in V(k) by conjugation T — > gTg^ 1 , the characteristic 
polynomial of T is an invariant of its orbit. Since T has trace zero, this monic characteristic polynomial 
has the form 

f(x) = det{xl — T) = x 2n+1 + c 2 x 2n ~ l + c 3 x 2n ~ 2 + ■■■ + c 2n+1 (6) 

with coefficients c m E k. The coefficients c m are polynomial invariants of the representation, with 
deg(c m ) = m. These invariants are algebraically independent, and generate the full ring of polynomial 
invariants on V (cf. [3]). An important polynomial invariant, of degree 2n(2n + 1), is the multiple 

A = A(c 2 , c 3 , . . . , c 2 „+i) = 4 2n disc/(x) 

of the discriminant of the characteristic polynomial of T, which gives the discriminant A of the asso- 
ciated hyperelliptic curve y 2 = f(x) (see [33]). The discriminant A(T) is non-zero precisely when the 
polynomial f(x) is separable over k, so has 2n + 1 distinct roots in a separable closure k s . 

Since the characteristic polynomial is an invariant of the G(A;)-orbit, it makes sense to describe the 
set of orbits with a fixed characteristic polynomial f(x). We begin by showing that, for any separable 
f(x), there is a naturally associated orbit of G(k) on V(k), which we call the distinguished orbit, that 
has associated characteristic polynomial f(x). 
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4.1 The distinguished orbit 

We assume throughout that f(x) is separable (i.e., disc(/) 7^ 0) and is of the form (6). 

Proposition 6 The group G(k) acts simply transitively on the pairs (T, X), where T is a self-adjoint 
operator in V(k) with characteristic polynomial f(x) and X C W(k) is a maximal isotropic subspace 
of dimension n with T(X) C X- 1 . 

The action of g on pairs (T, X) is by the formula (gTg~ 1 ,g(X)). As a corollary, we can identify 
a distinguished orbit of G(k) on the set of self-adjoint operators with characteristic polynomial f(x), 
namely, the orbit consisting of those T for which a maximal isotropic subspace X exists over k satisfy- 
ing T(X) C X- 1 . We will frequently use S to denote a self-adjoint operator in this distinguished orbit. 

Next, let L = k[x]/ (f(x)). Since we have assumed that f(x) is separable, L is an etale A;-algebra 
of rank 2n + 1. 

Proposition 7 The stabilizer Gs of a vector S in the distinguished orbit with characteristic polynomial 
f(x) is (uniquely) isomorphic to the finite commutative etale group scheme D of order 2 2n over k, 
which is the kernel of the norm map ReSL/kifa) ~* A*2- 

To prove these results, we need a method to construct orbits over k with a fixed separable char- 
acteristic polynomial f(x). Let be the image of x in L, so a power basis of L over k is given by 
{1, (3, (3 2 , . . . , (3 2n }. We define a symmetric bilinear form ( , ) on L by the formula 

(A, v) := the coefficient of (3 2n in the product \v. (7) 

Since fix) is separable, it is relatively prime to the derivative f'(x) in k[x] and the value f'((3) is a unit 
in L*. We then have another formula for this inner product which due to Euler [44, Ch III, §6]: 

(A,v)=Trace L/fc (Av//'(/3)). 

The orthogonal space we have defined on L is non-degenerate, of dimension 2n+l and discriminant 
= 1 over k. Indeed, the entries of its Gram matrix with respect to the power basis are above the 
anti-diagonal, and 1 on the anti-diagonal. We next observe that the A;-subspace M of L spanned by 
{1, (3, (3 2 , . . . , /3 n_1 } has dimension n and is isotropic for the bilinear form, so the orthogonal space L 
is split [30]. Hence we have an isometry 9 : L — > W(k) defined over k. This isometry is well-defined 
up to composition with an orthogonal transformation g of W(k) — any other isometry has the form 
9' = g9 where g is an element of O(W) = SO(W) x (±J). 

The fc-linear map (3 : L — > L given by multiplication by (3 is a self-adjoint transformation with 
characteristic polynomial equal to f(x). Hence the operator T = 9f39~ l : W(k) — > W(k) is self- 
adjoint with the same characteristic polynomial. If we compose the isometry 9 with the orthogonal 
transformation g, we obtain an operator gTg^ 1 in the same G(k) = SO(W)(k) orbit. This follows 
from the fact that the dimension of W is odd, so the orthogonal group and its special orthogonal 
subgroup have the same orbits on the representation V. 

This orbit we have just constructed is distinguished, as (3(M) is contained in M L = {1, f3, . . . , (3 n }. 
Hence, X = 9(M) is a maximal isotropic subspace of W over k with T(X) C X^, and (T, X) is a 
distinguished pair in V, defined over k. 
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To see that G(k) acts simply-transitively on the pairs (T, X), we first show that this is true when the 
field k = k s is separably closed. Since f(x) is separable, it is also the minimal polynomial of T on W . 
By the cyclic decomposition theorem, the centralizer of T in End(W) is the algebra k[T] generated 
by T, which is isomorphic to L. When k = k s is separably closed, this algebra is isomorphic to the 
product of 2n + 1 copies of the field k. If T' has the same characteristic polynomial, then T' = hTh~ l , 
where h E GL(W)(k) is well-defined up to right multiplication by (k[T])*. Since both operators T, T' 
are self-adjoint, the composition h*h centralizes T, so a = h*h lies in (k[T})*. In the separably closed 
case, every unit in the algebra k[T] is a square, so we may modify h by a square root of a to arrange 
that h*h = 1. Modifying h by ±1 if necessary, we may also arrange that det(h) = 1. It follows that T 1 
is in the same G(k) orbit as T. 

The subgroup Gxik) stabilizing T is the intersection of two subgroups of GL(W)(k). The first is 
the centralizer (k[T})*, where all the operators are self-adjoint (g* = g). The second is the subgroup 
SO(W)(k), where all the operators are special orthogonal (g* = g~ x , det(g) = 1) . Hence Gr(k) is 
isomorphic to the group 

D = {g G L* : g 2 = 1, N(g) = 1}. 

Since L is the product of 2n + 1 copies of k, the group D is an elementary abelian 2-group of 
order 2 2n . For future reference, we record what we have just proved. 

Propositions When k = k s is separably closed, the group G{k) = SO(W)(k) acts transitively on 
the set of self-adjoint operators with a fixed separable characteristic polynomial f(x). The stabilizer 
Gxik) of an operator T in this orbit is an elementary abelian 2-group of order 2 2n . 

We remark that GT(k s ) embeds as a Jordan subgroup of SO(W) (k s ), stabilizing a decomposition of 
W into a direct sum of orthogonal lines. This subgroup is unique up to conjugacy, and is not contained 
in any maximal torus. It is its own centralizer, and its normalizer is a finite group isomorphic to the 
Weyl group W(D 2n +i) = 2 2n .S 2n +i, where S 2n +i is the symmetric group (see [31]). 

We continue with the proof of Proposition 6 in the case when k is separably closed. The finite 
stabilizer Gr(k) acts on the set of maximal isotropic subspaces X in V with T(X) contained in X^, 
and we must show that it acts simply transitively. There are precisely 2 2n such subspaces X defined 
over an algebraic closure of k: these are the points of the Fano variety of the complete intersection of 
the two quadrics 

Q(w) = {w,w), Q T (w) = (w,Tw) 

in P(W). For an excellent discussion of the Fano variety of the complete intersection of two quadrics 
over the complex numbers, see [21]. This material was also treated in the unpublished Cambridge 
Ph.D. thesis of M. Reid [41]. A full treatment of the arithmetic theory sketched below will appear in 
the Harvard Ph.D. thesis of X. Wang [51]. 

The pencil xQ — x'Qt is non-degenerate. Using the matrices in our standard basis of W we have 
the formula 

disc(xA-B) = (— l) n det(xA — AM) = (-l) n det(A) det(xl — M) = f(x) 

and f(x) is separable. To show that these subspaces X of W are all defined over the separably closed 
field k, and that Gx(k) acts simply-transitively on them, we consider one such isotropic subspace in 
the model orthogonal space L, with form (A, v) given by (7) and the self-adjoint operator given by 
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multiplication by (3. The subspace M with basis {1, (3, . . . , f3 n ~ 1 } is maximal isotropic, and f3(M) is 
contained in M ± . The subgroup of L* which fixes M under multiplication is clearly k*. Hence the 
subgroup of D which fixes M is trivial and the finite group D must act simply-transitively on the 2 2n 
such subspaces (as it has the same order). This also shows that all of the 2 2n points of the Fano variety 
are defined over the separably closed field k. Translating this to W via the isometry 9 completes the 
proof of Proposition 6 when the base field k is separably closed. 

For general fields k, we use Galois descent. Let k s denote a separable closure of k and let Gal(k s /k) 
denote the Galois group. Consider two distinguished pairs (T, X) and (T", X') which are defined over 
k. By the simple transitivity result we have just proved over k s , there is a unique element g in the group 
SO(W)(fc a ) with g(T, X) = (T", X'). For any o in GaA(k s /k), the element g° has the same property. 
Hence g = g u lies in SO(W)(k), which completes the proof of Proposition 6 for general k. 

To determine the stabilizer of a vector in the distinguished orbit, as a finite group scheme over k, 
we again may use the model orthogonal space L with inner product (A, v) given by (7) and the distin- 
guished self-adjoint operator given by multiplication by (3. The centralizer D of (3 has points 

D(E) = {ge{L® E)* :g 2 = l, N(g) = 1}. 

in any A;-algebra E. Hence D = Res L / fc (/i 2 ) n=i is the finite group scheme over k described in Propo- 
sition 7. The choice of isometry 9 : L — > W gives a vector S = 9f39 v in the distinguished orbit and 
an isomorphism of stabilizers Gs = 9D9~ X . Since 9 is well-defined up to left multiplication by Gs, 
which is a commutative group scheme, the isomorphism with D is uniquely defined. This completes 
the proof of Proposition 7. 

Here is a condition on the Gram matrix B of the bilinear form (v, Tw) which implies that the orbit 
of T is distinguished. 

Proposition 9 An orbit of SO(W)(k) on the representation V{k) is distinguished if and only if it 
contains a vector T whose Gram matrix B = AM with respect to the standard basis has the form 

/0 •■• * *\ 
■ • • * * * 





* 

* * 

Proof: If the Gram matrix B of T has the form (8), then the subspace X spanned by the basis el- 
ements ei, &2, ■ ■ ■ , e n is isotropic and T(X) is contained in X L , which is spanned by the elements 
ei,&2,..., e n , u. Hence the orbit is distinguished. 

Conversely, if the operator T lies in a distinguished orbit, we may identify W with the space 
L = k[x]/(f(x)) having bilinear form (A, v) given by (7), via an isometry 9 : L — >■ W. The self-adjoint 
operator T = 9f39~ l has Gram matrix of the form (8) with respect to the basis 1, (3, (3 2 , . . . , (3 2n of L. It 
also has this form with respect to the basis 1, Pi(/3),p2((3), • • • , P2n{@) for any monic polynomials Pi(x) 
of degree i. But we can choose the monic polynomials Pi(x) so that the inner products (Pi((3),Pj((3)) 



* 


* 


* 


* 


* 


* 


* 


* 


*} 
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on L match the inner products of our standard basis elements on W. There is then an isometry 6' = 
g6 : L — >■ V taking this basis to our standard basis, and the vector 9' (39'~ l = gTg~ l in the orbit of T 
has Gram matrix B of the desired form. □ 



4.2 The remaining orbits via Galois cohomology 

To describe the remaining orbits with a fixed separable characteristic polynomial, we use Galois co- 
homology. Let k s denote a separable closure of k. If M is a commutative finite etale group scheme 
over k we let H l (k, M) = H l (Gal(k s /k), M(k s )) be the corresponding Galois cohomology groups. 
Similarly, if J is a smooth algebraic group over k, we let H l {k, J) = H 1 (Gs&(k a /k), J(k s )) be the 
corresponding pointed set of first cohomology classes [43, Ch I, §5]. If we have a homomorphism 
M — > J over k, we obtain an induced map of pointed sets H l (k, M) — > H l {k, J). The kernel of this 
map is by definition the subset of classes in H 1 ^, M) that map to the trivial class in H l (k, J). 

Let S be a self-adjoint operator with characteristic polynomial f{x) in the distinguished orbit of 
G = SO(W) over k, and let G s be the finite etale subgroup stabilizing S. Let T be any self-adjoint 
operator with this characteristic polynomial over k. By Proposition 8, the vector T is in the same orbit 
as S over the separable closure k s , so T = gSg^ 1 for some g E SO(W)(k s ). For any a E Gal(k s /k), 
the element c a = g~ x g a lies in the subgroup Gs(k s ), and the map a — > c CT defines a 1-cocycle on the 
Galois group with values in Gs(k s ), whose cohomology class depends only on the G(A;)-orbit of T. 
This class clearly lies in the kernel of the map 

V :H 1 (k,G s )^H 1 (k,G). (9) 

Conversely, if we have a cocycle c a whose cohomology class lies in the kernel of rj, then it has the 
form g~ x g a and the operator T = gSg~ x is defined on W over k. Since T has the same characteristic 
polynomial as S, we have established the following. 

Proposition 10 The map which takes an operator T to the cohomology class of the cocycle c a induces 
a bijection between the set of orbits ofG(k) on self-adjoint operators T with characteristic polynomial 
f(x) and the set of elements in the kernel of the map r/ : H 1 ^, Gs) —> H l {k, G). The trivial class in 
H l {k, Gs) corresponds to the distinguished orbit. 

We will make the map rj more explicit shortly, via a computation of the pointed cohomology sets of 
Gs and G. But we can use our discussion of the Fano variety to give an explicit principal homogeneous 
space Pt for the finite group scheme Gs over k, corresponding to the self-adjoint operator T. Choose 
g E G{k s ) / Gs{k s ) with gSg^ 1 = T. Since Gs is abelian we obtain a unique isomorphism of the 
stablizers Gs — Gt over k s , which is given by conjugation by any g in this coset. The unicity implies 
that this isomorphism is defined over k. It therefore suffices to construct a principal homogeneous 
space Pt for the finite group scheme Gt- We let P T be the set of maximal isotropic subspaces X cW 
defined over k s with the property that T{X) C X^. This set is finite, of cardinality 2 2n . The Galois 
group Gal(k s /k) acts, so Pt has the structure of a reduced scheme of dimension zero over k and forms 
a principal homogeneous space for the group scheme Gt- The isomorphism class of the principal 
homogeneous space Pt over k depends only on the orbit of T, and corresponds to the cohomology 
class of the cocycle c a . Note that the scheme Pt has a /^-rational point if and only if T lies in the 
distinguished orbit. 
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We now make the above map 77 of pointed sets more explicit, using our model orthogonal space L 
with form (A, v) given by (7). First, since 2n + 1 is odd, we have a splitting 

Res L/fc (/i 2 ) = n 2 © D. 

This allows us to compute the Galois cohomology groups of D using Kummer theory. We find that 

H°(k,D) = L*[2] N=1 

H l (k,D) = (L* /L* 2 ) N =i 

where we use the subscript N = 1 to mean the subgroup of elements in L* / L* 2 whose norms are 
squares in k*. 

Let a be a class in H x (k, D), viewed as an element of the group L* whose norm to k* is a square. 
The class 77(a) lies in the pointed set H 1 ^, SO(L)), which classifies non-degenerate orthogonal spaces 
of the same dimension 2n + 1 and discriminant = 1 as our model space L over k (see [32]). Unwinding 
the definitions as in [3], we find that the orthogonal space corresponding to the class 77(a) is isometric 
to L with the modified bilinear form (A, v) a = the coefficient of (3 2n in the product a\v. Equivalently, 
we have the formula: 

(A,i/) a = Trace L/fc (aAz///'(/3)). 

We note that the isomorphism class of this space depends only on the element a in the quotient group 
(L* / L* 2 ), and that the condition that the norm of a is a square in k* translates into the fact that the 
discriminant of the form (A, u) a is = 1. 

If we translate this over to the orthogonal space W via the isometry 9, we see that a class a in 
the group H\k,G s ) = H l (k,D) = (L*/L* 2 ) Nsl lies in the kernel of the map 77 : H\k,G s ) -> 
H x (k, SO(W)) precisely when the form (A, v) a on L gives a split orthogonal space over k. In that case 
a gives an orbit of self-adjoint operators T on W, and the principal homogeneous space Pt splits over 
the extension of L where a is a square. We will produce examples of non-trivial a where the bilinear 
form (A, v) a is split in the next section, using the Jacobians of hyperelliptic curves. 

5 Hyperelliptic curves 

We now use some results in algebraic geometry, on hyperelliptic curves and the Fano variety of the 
complete intersection of two quadrics in F(W © k), to produce classes in the kernel of the map in 
cohomology 77 : H l {k, Gs) — > H l {k, SO(W)), and hence orbits with a fixed separable characteristic 
polynomial f(x). 

Let C be the smooth projective hyperelliptic curve of genus n over k with affine equation y 2 = f(x) 
and a A;-rational Weierstrass point O above x = 00. The functions on C which are regular outside of P 
form an integral domain: 

H°(C - P, C -o) = k[x,y}/(y 2 = f(x)) = k[x, y/f(xj]. 

The complete curve C is covered by this affine open subset U±, together with the affine open subset 
U2 associated to the equation w 2 = v 2n+2 f(l/v) = C2 n +iv 2n+2 + • • • + C2V 3 + v and containing the 
point O = (0, 0). The gluing of TJ\ and U 2 is by (v, w) = (1/x, y/x n+1 ) and (x, y) = (1/u, w/v n+l ) 
wherever these maps are defined. 
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5.1 The 2-torsion subgroup 

Let J = Pic (C) denote the Jacobian of the hyperelliptic curve C over k, which is an abelian variety of 
dimension n over k, and let J [2] denote the 2-torsion subgroup of J. This is a finite etale group scheme 
of order 2 2n . Recall that we have defined a finite group scheme D = (Res£/fc/x 2 )jv=i of this order, 
associated to the etale algebra L = k[x}/ (f(x)). We begin by constructing an isomorphism D = J [2] 
of finite group schemes over k. 

The other Weierstrass points P 7 ^ O of C(k s ) correspond bijectively to fc-algebra homomorphisms 
7 : L — > k s : we have x(Ry) = j{/3) and y{P 1 ) = 0. Associated to such a point we have the divisor 
d 7 = (P 7 ) — (O) of degree zero. The divisor class of d 7 lies in the 2-torsion subgroup J[2](k s ) of the 
Jacobian, as 2(d 7 ) is the divisor of the function (x — y(/3)). The Riemann-Roch theorem shows that 
the classes <i 7 generate the finite group J[2](k s ), and satisfy the single relation XX ^7) = 0- Indeed, 
this sum is the divisor of the function y. This gives an isomorphism Res L / fc (/i 2 ) / ^2 — J[2] ■ We recall 
that D is defined as is the kernel of the map Res L / fc (/i 2 ) — > /i 2 . Since the degree of L over k is odd, 
the composition D — > ResL/fc(/i 2 ) — > ResL/ / t(/i 2 )//i 2 is an isomorphism. So we have constructed an 
isomorphism from D to J [2]. 

The exact sequence — > J [2] — > J — > J — > of commutative group schemes over k s gives an 
exact descent sequence in Galois cohomology: 

->■ J(k)/2J(k) -> H\k, J[2\) -> if 1 ^, J) [2] ->■ 0. 

The middle term H x {k, J[2]) can be identified with the group D) = (L* / 'L* 2 )n=i via the above 

isomorphism. We need an explicit description of the coboundary map 

5 : J(k)/2J(k) -)• (L* / L* 2 ) N =i. 

If P = (a, 6) is a point of C over with 6 7^ 0, and D = (P) — (O) is the corresponding divisor class 
in J(jfe), then (cf. [42], [48, p.7]) 

<J(D) = (a-/3). 

5.2 A subgroup in the kernel of 77 

For any self-adjoint operator S in the distinguished orbit, we have constructed an isomorphism from 
D to the stabilizer Gs in SO(W). Hence we may identify the cohomology groups H 1 ^, J[2]) = 
H l {k, Gs). With this identification, we have the following important result. 

Proposition 11 The subgroup J(k) /2J(k) of H 1 ^, J [2]) lies in the kernel of the map rj : H l {k,G s ) — > 
H\k,SO(W)). 

Thus classes a in the subgroup J{k) /2J{k) correspond to orbits of SO(W)(k) on V(k) with charac- 
teristic polynomial f(x). 

Proof: To prove this proposition, we need to make the descent map from H l {k, D) to H l {k, J)[2] 
more explicit. That is, we need to associate to a class a in the group H l {k,D) = (L*/L* 2 ) N =i a 
principal homogeneous space F a of order 2 for the Jacobian J over k. The class a will lie in the 
subgroup J(k)/2J(k) precisely when the homogeneous space F a has a A;-rational point. We will use 
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the existence of this /c-rational point to show that the bilinear form ( , ) a on L is split, so a corresponds 
to a class in the kernel of 77. 

More generally, let a be any class in H x {k,D) = (L* / L* 2 )n=i- The vector space L © k has 
dimension 2n + 2 over /c. Consider the two quadrics on P(L © fc): 

Q(A,a) = (A, A) Q = Trace(«A 2 /f (/?)), 

Q'(A, a) = (A, /3A) Q + a 2 = Trace(a/3A 2 /f (/?)) + a 2 . 

The pencil uQ — vQ' is non-degenerate and contains exactly 2n + 2 singular elements: the quadric Q 
at v = and the 2n + 1 quadrics 7Q — Q' at the points (7, 1) where 7(7) = 0. More precisely, the 
discriminant of the pencil is given by the binary form 

disc(wQ - vQ') = v 2n+2 f(u/v) 

of degree 2n + 2 over k. Hence the base locus Q fl Q' of the pencil is non-singular. The Fano 
variety F = F a of this complete intersection, consisting of the n-dimensional subspaces Z of the 
2n + 2 dimensional space L © k which are isotropic for all of the quadrics in the pencil, is smooth of 
dimension n (see [21]). 

Since the discriminant of the the pencil defines the hyperelliptic curve C over k, a point c = (x,y) 
on the curve determines both a quadric Q x = xQ — Q' in the pencil together with a ruling of Q x . (A 
ruling of Q x is, by definition, a component of the variety of n + 1 dimensional subspaces in L © fc 
which are isotropic for Each point of C gives an involution of the corresponding Fano variety 
r(c) : F -> F, which has 2 2n fixed points over a separable closure k s , and is defined as follows. 
A point of F over k s consist of a common isotropic subspace Z of dimension n for all quadrics in 
the pencil. The point c then gives a fixed maximal isotropic subspace Y of dimension n + 1 for the 
quadric Q x which contains Z. If we restrict any non-singular quadric other than Q x in the pencil to the 
subspace Y, we get a reducible quadric which is the sum of two hyperplanes: Z and another common 
isotropic subspace r(c)(Z). This defines the involution, and hence a morphism <\> : C x F — >• F with 
0(c,Z)=t(c)(Z). 

The morphism C x F — >■ F can be used to define a fixed-point free action of the Jacobian J = 
Pic°(C) on F over k, which gives F the structure of a principal homogeneous space for J. For a 
general non-degenerate pencil of quadrics, this principal homogeneous space has order 4 in the group 
J), and its square is the principal homogeneous space Pic 1 (C). In this case, we have a A;-rational 
Weierstrass point O on C, which corresponds to the degenerate quadric Q in the pencil. Hence Pic 1 (C) 
has a rational point and the homogenous space F a has order 2 in H l {k, J). 

We can prove that F a represents the image of a in the descent sequence as follows. The involution 
r = t(0) is induced by the linear involution (A, a) — > (A, —a). Hence the fixed points P a of r are just 
the n-dimensional subspaces X of L over k s which are isotropic for both quadrics Q and Q' . This is 
the Fano variety P T = P a of dimension zero, which is the principal homogeneous space for Gt = J[2] 
associated to the cohomology class a. It follows that F a is the image of a in H l {k, J)[2]. For more 
details on this construction, see the Harvard Ph.D. thesis of X. Wang [51]. Note that the variety F a has 
a k -rational point whenever a lies in the subgroup J(k)/2J(k), but F a only has a A;-rational point fixed 
by the involution r when a is the trivial class in H l {k, J [2]). 

We may now prove Proposition 11. Assume that the class a is in the subgroup J(k)/2J(k). Then 
its image in H 1 (k, J) [2] is trivial, and the homogenous space F a has a fc-rational point. Hence there is a 
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^-subspace Z of L © k which is isotropic for both Q and Q'. Since it is isotropic for Q', the subspace Z 
does not contain the line 0@k. Its projection to the subspace L has dimension n and is isotropic for Q. 
This implies that the orthogonal space L with bilinear form (A, u) a = Trace(o;Ai/// / (/3)) is split, so 
the class a is in the kernel of the map 7]. □ 

Remark 12 We note that the finite group scheme D = J [2] does not determine the hyperelliptic 
curve C. Indeed, for any class d E k* /k* 2 , one has the hyperelliptic curve Cd with equation dy 2 = 
f(x) and Jacobian Jd. The determination of J [2] shows that these Jacobians share the same 2-torsion 
subgroup: J d [2] = J[2]. This Jacobian Jd acts simply transitively on the Fano variety F a ^ of the 
complete intersection of the two quadrics 

Q(X,a) = Trace(aA 2 /f (/?)), 

Q'(A, a) = Trace(a/3A 2 //'(/3)) + da 2 . 

Indeed, the discriminant of the quadric uQ — vQ' in the pencil is equal to dv 2n+2 f [u / v) . A similar 
argument to the one given in the proof of Proposition 11 shows that the subgroup J d (k)/2J d (k) of 
H x {k, D) is also contained in the kernel of the map r\ : H 1 {k, D) ->■ H x (k, SO(W)). 

6 Orbits over arithmetic fields 

In this section, we describe the orbits with a fixed characteristic polynomial, when A; is a finite, local, 
or global field. 

6.1 Orbits over a finite field 

We first consider the case when k is finite, of odd order q. In this case, every non-degenerate quadratic 
space of odd dimension is split, so SO(W)) = 1, and the orbits with a fixed separable charac- 

teristic polynomial f(x) are in bijection with elements of the group H 1 ^, D) = (L* / L* 2 ) N =i. This 
elementary abelian 2-group has order 2 m , where m + 1 is the number of irreducible factors of f(x) in 
k[x\. This is also equal to the order of the stabilizer H°(k, D) = D{k) = L*[2]yv = i of any point in the 
orbit over k. Hence the number of vectors T in V with any fixed separable polynomial is equal to the 
order of the finite group SO (W)(q), and is given by the formula 

#so(w)(?) = <f V" - i)(? 2n ~ 2 - 1) • ■ ■ (q 2 - !)• 

Since k is perfect, a similar argument applies to all regular orbit — those orbits for which the char- 
acteristic polynomial of T is equal to its minimal polynomial. Any regular vector T with characteristic 
and minimal polynomial f(x) has stabilizer D, the finite etale group scheme which is the kernel of the 
map Res£/ fc /i 2 — > yU 2 with L = k[x]/(f(x)). This finite group scheme has order 2 r over k, where r + 1 
is the number of distinct roots of f(x) in the separable closure k s . (Here < r < 2n, as we do not 
assume that the minimal polynomial f(x) is separable.) Since the number of regular orbits with this 
characteristic polynomial is the order of the finite group D), and this order is equal to the order 

of the stabilizer H°(k, D) of any point in the orbit, the number of regular vectors T with any fixed 
characteristic polynomial f(x) is equal to the order of the finite group SO(W)(q). Since there are q 2n 
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choices for the coefficients of the characteristic polynomial f(x), we can count the total number of 
regular vectors in V(q). This number is 

q 2n . #SO(W)(q) = q d - q d ~ 2 + ■■■ 

where d = dim V. Since equality of the characteristic and minimal polynomials is an algebraic condi- 
tion, this count shows that irregularity is a codimension 2 condition. 

We return to the case where f(x) is separable. By Lang's theorem, we have H 1 ^, J) = 0, where J 
is the (connected) Jacobian of the smooth hyperelliptic curve C with equation y 2 = f(x) over k. Hence 
the homomorphism J{k) /2J{k) — > H 1 ^, D) is an isomorphism, and every orbit with characteristic 
polynomial f(x) comes from a A;-rational point on the Jacobian. 

6.2 Orbits over a non-archimedean local field 

Next, we consider the case when k is a non-archimedean local field, with ring of integers A and 
finite residue field A/tvA. In this case, the spin cover gives a connecting homomorphism in Galois 
cohomology, which is an isomorphism by Kneser's theorem [28], cf. [37, Thms 6.4, 6.10, pp. 356- 
397]: 

H\k, SO(W0) = H 2 {k, fr) = (Z/2Z). 

One can show that the resulting map r\ : D) — > H x (k, SO(W / )) = (Z/2Z) is an even quadratic 

form whose associated bilinear form is the cup product on H x (k, J [2]) induced from the Weil pairing 
J[2] x J [2] — > /i 2 [51], cf. [39]. The subgroup J(k)/2J(k) is a maximal isotropic subspace on which 
r\ — 0. This allows us to count the number of orbits with a fixed separable characteristic polynomial 

/(*)■ 

We first do this count when the characteristic of the residue field is odd. Let m + 1 be the number 
of irreducible factors of f(x) in k[x], and let A L be the integral closure of the ring A in L. Then the 
stabilizer H°(k, D) = D(k) has order 2 m . This is the order of H 2 (k, D) by Tate's local duality theo- 
rem, and his Euler characteristic formula then shows that the group H l {k, D) has order 2 2m (see [43, 
§11.5]). The number of orbits with characteristic polynomial f(x) is equal to the number of zeroes of 
the even quadratic form 77 on this vector space, which is 2 m_1 (2 m + 1) = 2 2m_1 + 2 m ~ x . The subgroup 
J(k)/2J(k) has order 2 m ; it is a maximal isotropic subspace for this quadratic form. We note that 2 m 
is also the order of the subgroup (A* L /A* 2 ) N =x of units in (L* / L* 2 ) N =x. We will see that these two 
subgroups of (L* / L* 2 ) n=i coincide in many cases, such as when the polynomial f(x) has coefficients 
in A and the order A[x]/ (f(x)) is equal to the integral closure Al of A in L. 

Next assume that the characteristic of the residue field is even, and for simplicity that k is an 
unramified extension of Q2. Let m + 1 be the number of irreducible factors of f(x) in k[x], and let At 
be the integral closure of the ring A in L. Then the stabilizer H°(k, D) = D(k) has order 2 m . This is 
the order of H 2 (k, D) by Tate's local duality theorem, and his Euler characteristic formula now shows 
that the group H l {k, D) has order 2 2m+2n . The number of orbits with characteristic polynomial f(x) 
is again equal to the number of zeroes of the even quadratic form 77 on this vector space. The subgroup 
J(k)/2J(k) has order 2 m+n ; it is a maximal isotropic subspace for the quadratic form. Here the order 
of the subgroup (A* L /A* 2 ) N =i of units in (L* / L* 2 ) N =i is equal to 2 m+2n , which is strictly larger than 
the order of J{k)/2J{k). 
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6.3 Orbits over C and R 



The local field k = C is algebraically closed, so there is a unique (distinguished) orbit for each separa- 
ble characteristic polynomial by Proposition 8. 

The classification of orbits is more interesting for the local field k = R. We now work this out in 
some detail, as we will need an explicit description of the real orbits in order to use arguments from 
the geometry of numbers later in this paper. Over the real numbers the pointed set if 1 (IR, SO(W)) has 
+ 1 elements, corresponding to the quadratic spaces of signature (p, q) with p + q = 2n + 1 and 
q = n (mod 2). The split space W of dimension 2n + 1 and discriminant = 1 has signature (n + 1, n) 
overK, so SO(W) = SO(n + l,n). 

Let 2m + 1 be the number of real roots of f(x), with < m < n. Then the polynomial f(x) has 
exactly (2m + 1) + (n — m) = m + n + 1 irreducible factors over R, and L = IR 2m+1 x C n_m . The 
stabilizer of any self-adjoint operator T with this characteristic polynomial is an elementary abelian 2- 
group of order 2 n+m , isomorphic to H°(R, D) = L*[2] N=1 . The group H^R, D) = ((R* /R* 2 ) 2rn+1 x 
(C*/C* 2 ) n ~ m )yv=i is an elementary abelian 2 group of order 2 2m . The real locus of the hyperelliptic 
curve y 2 = f(x) has m + 1 components, and the group J(R)/2J(R) = J(R)/J(R)° has order 2 m . 
This is a maximal isotropic subspace of J [2]) for the cup product pairing and each class in 

J(R)/2J(R) gives a soluble real orbit for the split group. 

There are ( 2r ^ fl ) orbits of SO(n + 1, n) on the self-adjoint operators T with characteristic poly- 
nomial f(x). One can identify these orbits as follows. Such an operator T has 2m + 1 distinct real 
eigenspaces of dimension 1 and (n — m) stable subspaces of dimension 2, which correspond to the pairs 
of complex conjugate eigenvalues. The orthogonal space W(R) decomposes as an orthogonal direct 
sum of these eigenspaces, and the 2-dimensional eigenspaces each have signature (1, 1). It follows that 
the orthogonal sum of the 1 -dimensional eigenspaces has signature (m + 1, m), and the subset of m 
eigenspaces which are negative definite determines the real orbit of T. 

Let 

Ao < Ai < ■ ■ ■ < 

be the real roots of f(x). The operators S in the distinguished orbit have eigenvectors S(wi) = \i.Wi 
with (wi, Wi) = (—1)*. In other words, the signs of the inner products of the eigenvectors of an operator 
in the distinguished orbit is 

+ - + - ••• + (10) 

Using the explicit description of the map 5 : J(R)/2J(R) — > J[2]), we see that for a point 

P = (a, b) on the curve, with X 2 k < a < A 2fc+ i, the class of the divisor D = (P) — (O) gives an orbit 
with eigenvectors w.i satisfying (lOj, = (— I) 1 for % < 2k and (wi, w^) = (— for i > 2k + 1. It 
follows that the signs of the inner products of eigenvectors of an operator in a soluble real orbit have 
the form 

+ ei e 2 • • • e m (11) 

where each ^ is equal to either ( — h) or (H — ). 

The space R 2n — {A = 0} of characteristic polynomials with non-zero discriminant has n + 1 
components I[m) indexed by integers < m < n, where 2m + 1 is the number of real roots. We get 
a similar decomposition of the space V(R) — {A = 0} of self-adjoint operators T with characteristic 
polynomials in I[m). We label the various components V^ m,T ^ in V(R) which map to the component 
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J(m) in IR 2n — {A = 0} by integers 1 < r < ( 2m+ "j , and let V(m, 1) be the set of operators which lie 
in the distinguished orbits. 

The integer r indexes a choice of signs (wi, wi) = ±1 for the inner products of the 2m + 1 real 
eigenvectors Wi of T, with m signs equal to —1. We use n to denote the distinguished orbit, where 
we have seen that the signs for the inner products, ordered for increasing real eigenvalues, are given 
by (10). We use the first 2 m indices ti,t 2 , • ■ ■ , t 2 ™ to denote the soluble orbits, whose signs are 
described in (1 1). Note that the stabilizer of every vector T in the component lA m,T ) has order 2 m+n , 
independent of r. 

6.4 Orbits over a global field 

Finally, we consider the case when A; is a global field. In this case the group H l {k, D) = H l {k, J [2]) 
is infinite. We will see why there are also infinitely many classes in the kernel of 77, so infinitely many 
orbits of self-adjoint operators with characteristic polynomial f(x). First, we show that every class in 
the 2-Selmer group lies in the kernel of 77. 

Proposition 13 Every class a in the 2-Selmer group S 2 (J) C H 1 ^, J[2}) lies in the kernel of r\, so 
corresponds to an orbit of self-adjoint operators over k with characteristic polynomial f(x). 

Proof: Let a be a class in the 2-Selmer group. By definition, these are the classes in H l {k, J[2}) whose 
restriction to H l {k v , J [2]) is in the image of J(k v )/2J(k v ) for every completion k v . By Proposition 11, 
the orthogonal space U v associated to the restriction of the class a in H l {k v , SO(W)) is split over k v , 
for every valuation v of k. By the theorem of Hasse and Minkowski, a non-degenerate orthogonal 
space U of dimension (2n + 1) is split over k if and only if U <E> k v is split over every completion k v . 
Hence the orthogonal space U associated to a is split over k, and a lies in the kernel of 77. □ 

We call the orbits corresponding to the 2-Selmer group locally soluble, as the Fano variety Ft 
associated to T has points over k v for all completions v. For each f(x), these locally soluble orbits are 
finite in number, as the 2-Selmer group is finite. 

We have already remarked that, for any class d G k* / k* 2 , the finite group scheme J[2] is isomorphic 
to the two torsion subgroup J d [2] of the Jacobian of the hyperelliptic curve Cd with equation dy 2 = 
f(x). Using similar methods, we can also obtain A;-rational orbits with characteristic polynomial f{x) 
from classes in the 2-Selmer group of Jd over k. Since the 2-Selmer groups of these quadratic twists of 
C become arbitrarily large finite elementary abelian 2-groups (see [1 1] for the case n = 1), the number 
of /c-rational orbits with characteristic polynomial f(x) is infinite. 

We have proven Theorem 5, and the remarks following it. 



7 Nilpotent orbits and Vinberg's theory 

We say that an orbit T of SO(W) on V is nilpotent if the characteristic polynomial of T is equal to 
f(x) = x 2n+l . We say that it is regular nilpotent if the minimal polynomial of T is also equal to x 2n+1 . 
One can show that the group SO(W) (k) acts simply-transitively on regular nilpotent elements in V(k), 
i.e., there is a single orbit with trivial stabilizer. A representative operator E on the standard basis is 
given by: 

E'fi->f2-> > fn -> u e n ->■ > e 2 ->■ ei ->■ 0. 
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We say that a nilpotent orbit T is sub-regular if the minimal polynomial of T is equal to x~ n . An 
example of a sub-regular nilpotent operator E' is given on the standard basis by: 

E''fi->f2-> > fn -> e n -> ► e 2 -> ei ->■ 0, u ->■ 0. 

The subregular nilpotent orbits of SO(W) (k) are indexed by the distinct classes d E k* / k* 2 and repre- 
sented by the operators d ■ E'. 

When the characteristic of k is either or is greater than 2n + 1, the representation V of SO(W) 
appears in Vinberg's theory of torsion automorphisms 9, where nilpotent orbits play a special role [36]. 
We review this connection briefly here, as a model for future work on those automophisms 9 which lift 
regular elliptic classes in the Weyl group. 

To do this, we change our notation for the rest of this section to conform with Vinberg, and let 
G = SL(W) with W of dimension 2n + 1 over k. This reductive algebraic group has Lie algebra 
g = {T e End(W) : Trace(T) = 0}. The outer (pinned) involution 9(g) = (g*)^ 1 has fixed subgroup 
G e = G(0) = SO(W) with Lie algebra g(0) = {T e End(V) : T + T* = 0}. The fixed subgroup 
SO(W) acts on the non-trivial eigenspace for 9 on the Lie algebra: 

(1) = {T e End(V) : T* = T, Trace(T) = 0} 

and this is precisely the representation V that we have been studying. The invariant polynomials for 
G(0) on in this case are the restriction of the invariant polynomials c 2 , C3, . . . , c 2n+ i for G on the 
adjoint representation g. This gives a polynomial map to the geometric quotient: 

p : 0(1) -> fl(l)//G(0) = Spec k[c 2 , . . . , c 2n+1 ]. 

The regular nilpotent vector E in g(l) can be completed to a principal st 2 -triple [E, F, H], with F 
regular nilpotent in g(l) and H regular semi-simple in g(0). Similarly, the subregular nilpotent vector 
E' in g(l) can be completed to a subregular s[ 2 -triple [E', F', H'\, with F' subregular nilpotent in g(l) 
and H' semi-simple in g(0). The centralizer z(F) of F in the Lie algebra g is abelian of dimension 2n 
and is contained in g(l). If we restrict the invariant polynomials to the affine subspace E + z(F) of 
g(l), the resulting map 

p : E + z(F) -> Spec k[c 2 , . . . , c 2n +i] 

is a bijection, whose inverse is called the Kostant section [36]. The isotropic subspace X of W spanned 
by {/1, / 2 , . . . , /„} satisfies the condition: T(X) is contained in X 1 - for all operators T in the Kostant 
section. It also contains no T-invariant subspace Y except for Y — 0. Hence the Kostant section gives 
a representative for each distinguished orbit, once a representative E of the regular nilpotent orbit has 
been chosen. 

In his Harvard PhD thesis, J. Thorne [50] studies the centralizer z(F') of a subregular nilpotent 
vector F' in g. This has dimension 2n + 2, and the eigenspace z(F')(l) has dimension 2n + 1 in 
g(l). The restriction of the invariant polynomials to the affine subspace E' + z(F'){l) of g(l) defines 
a family 

p: E' + z(F')(l) -)• Spec/c[c 2 ,...,c 2n+1 ] 

of affine curves over the geometric quotient, which can be identified with the affine hyperelliptic curves 
y 2 = f(x). They are smooth over the open subset where A ^ 0. If one starts instead with the subregular 
vector d ■ E' and forms the corresponding st 2 -triple, one gets the family of affine curves dy 2 = f(x), 
with the same 2-torsion in their Picard groups. 
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8 Integral orbits 



Fix a separable polynomial f(x) = x 2n+l + C2X 2n ~ 1 + c^x 2n ~ 2 + • • • + C2„+i over Q, let C be the 
hyperelliptic curve with equation y 2 = f(x) and let J be its Jacobian. We have seen that classes in 
the 2-Selmer group of J give locally soluble orbits for the split orthogonal group SO(W)(k) on V(k). 
However, since there are an infinite number of rational orbits and the Selmer group is finite, not all 
of the orbits over Q are locally soluble. In this section, we consider those orbits with an integral 
representative, in a sense to be defined below. We will show that this is a finite set which contains the 
locally soluble orbits. 

8.1 Orbits over Z p for p ^ 2 

We begin by discussing the integral orbits over Z p , with p ^ 2. In this case the group G = SO(W) is 
smooth over Z p , and the lattice W®Z P has rank 2n+l and discriminant disc(W) = (— l) n det(W) = 1 
in Z p /Z; 2 . 

Lemma 14 Assume that p ^ 2 and let I be a Z p -module of rank 2n + 1 equipped with a symmetric 
bilinear form I x / — > Z p whose discriminant is the square of a unit. Then I is isometric to W over Z p . 

This follows from the fact that there are, up to isomorphism, only two orthogonal spaces over 
Q p of dimension 2n + 1 and discriminant = 1. The non-split orthogonal space does not contain 
a non-degenerate lattice / over Z p , and the corresponding lattices in the split orthogonal space are all 
conjugate. The stabilizer of such a lattice is a hyperspecial maximal compact subgroup of SO(W)(Q p ). 

Using this lemma, we can give a simple classification of the orbits of SO(W)(Z p ) on the self- 
adjoint operators T : W <g> Z p — > W ® Z p with a fixed characteristic polynomial f(x) over Z p , which is 
assumed to be separable over Q p . Let R = Z p [x]/(f(x)), which is a Z p -order in the etale Q p -algebra 
L = Q p [x]/ (f(x)). We say a Z p -lattice I in L which spans L over Q p is a fractional ideal for R if it is 
stable under multiplication by R. Such an ideal / has a norm N(I), which is an element of Q*/Z*: if 
p a I is contained in R and the order of the finite Z p -module R/p a I is p b , then N(I) = p b ~ a . If a is an 
element of L*, we have the principal fractional ideal (a) consisting of all /^-multiples of a. This has 
norm = N(a). 

We will define an equivalence relation on pairs (/, a), where / is a fractional ideal for R, a is an 
element of L* whose norm is a square in Q* the square I 2 of the ideal / is contained in the principal 
ideal (a), and the square of its norm satisfies N(I) 2 = N(a) (mod Z* 2 ). We say that the pair (/', a') 
is equivalent to (/, a) if there is an element c in L* with I' = cl and a' = c 2 a. 

If R is maximal in L, every fractional ideal / is principal and every equivalence class is represented 
by a pair (R, a), where a is a unit having square norm to Z* In this case, the set of equivalence classes 
form a finite group (R* / R* 2 )n=i- hi general, the equivalence classes form a finite set which contains 
the finite group (R* / 'R* 2 )n=i corresponding to those pairs where the ideal / is principal. Since the 
ring R is generated by a single element (3 over Z p , it is Gorenstein of dimension one. Hence the ideal 
/ is principal if and only if it is proper, i.e., Endij(J) = R (see [6]). 

The set of equivalence classes always maps to the group (L* / L* 2 ) N =i, by taking the class of (/, a) 
to the class of a. This map need not be injective — for example, when the ring R is not maximal, a unit 
a that is not a square in R* may become a square in L*. 
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Proposition 15 Assume that p ^ 2 and that f(x) = x 2n+1 + c 2 x 2n ~ 1 + c 3 x 2n ~ 2 + • • • + c 2n +i is a 
polynomial with coefficients in Z p and non-zero discriminant. Then the integral orbits ofSO(W)(Z p ) 
on self-adjoint operators T in V(Z p ) with characteristic polynomial f(x) correspond to the equiva- 
lence classes of pairs (I, a) for the order R = Z p [x]/ (f(x)). The integral orbit corresponding to the 
pair (I, a) maps to the rational orbit of SO(W)(Q p ) on V(Q P ) corresponding to the class of a in 
(L* / L* 2 ) N =i. 

Proof: Starting with a pair (/, a) we construct an integral orbit as follows. Recall that L = Q p + 
Q p (3 + ■ ■ ■ + Q p [3 2n . Define the symmetric bilinear pairing / x / — >• 7h p by (A, z/) a -i := the coefficient 
of (3 2n in the product « _1 Aia Since the square I 2 of the ideal / is contained in the principal ideal (a), 
this product lies in the subring R = Z p + Z p (3 + • • • + Z p (3 2n and the coefficient of (3 2n is integral. 
Since N(I) 2 = N(a), this pairing has discriminant = 1. By the above lemma, the bilinear module / is 
isometric to W over Z p . Since / is an ideal of R, multiplication by f3 gives a self-adjoint operator on / 
with characteristic polynomial f(x). Translating this operator to W by the isometry gives the desired 
integral orbit. Since a self-adjoint T : W — > W gives W the structure of a torsion free Z P [T] = R- 
module of rank one, every integral orbit arises in this manner. Since the bilinear form defined above is 
equivalent to the form (A, u) a over Q p , it maps to the rational orbit of a. This completes the proof of 
Proposition 15. □ 

As a corollary, we obtain the following. 

Corollary 16 Assume that f(x) has coefficients in Z p and that the ring R = Z p [x]/(f(x)) is max- 
imal in L. Then the integral orbits correspond to classes a in the unit subgroup (R* / 'R* 2 )n=i of 
(L*/L* 2 ) n=i- Every rational orbit whose class lies in the unit subgroup has a unique integral repre- 
sentative. 

An important special case of the corollary is when the discriminant disc(/) of the characteristic 
polynomial is a unit in Z*. Then the hyperelliptic curve C with affine equation y 2 = f{x) over Z p has 
good reduction modulo p. The Jacobian J of C also has good reduction, and the finite group scheme 
J [2] is etale over Z p . It is obtained by restriction of scalars from the finite etale extension R of Z p , 
i.e., J[2] is isomorphic to the kernel of the map ReSi?/ Zp (/i 2 ) — > p,2- Taking the etale cohomology 
of the exact sequence of group schemes — > J\2] — > J — > J — > over Z p , we see that the image 
of J(Q P )/2J(Q P ) in H\Q P ,J[2}) = (L*/L* 2 )jv=i is precisely the unit subgroup H l (Z p , J[2\) = 
(R* /R* 2 )n=i- Hence we have obtained the following (which also holds under the weaker hypothesis 
that the order R is maximal in L). 

Corollary 17 Assume that f(x) has coefficients in Z p and that the discriminant of f (x) is a unit in Z*. 
Then the integral orbits with characteristic polynomial f(x) are are in bijection with the soluble orbits 
over Q p , i.e., those whose classes lie in the image of the subgroup J(Q P )/2J(Q P ). 

In general, given a p-adic rational orbit with integral characteristic polynomial f(x), we can ask if 
it is represented by an integral self-adjoint operator T : W <8> Z p — > W <g> Z p . That is, given an element 
a in L* with square norm to k*, which represents a class in the quotient in group (L* / L* 2 ) jv=i lying in 
the kernel of the map 77, we can ask if there an ideal I of R with I 2 C aR and (NI) 2 = N(a). One 
situation in which we can prove that an integral orbit exists is when the corresponding p-adic rational 
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orbit is soluble, i.e., for those elements a whose classes are contained in the subgroup J(Q p )/2 J(Q P ) 
of H l (® p ,J[2}) = (L*/L* 2 ) Nsl . 

We thank Michael Stoll for his help with the proof of the following key proposition: 

Proposition 18 Assume that f (x) has coefficients in Z p and non-zero discriminant. Let R = Z p [x]/(f (x) ) 
and L = Q p [x]/(f(x)). If a G L* represents a class in the subgroup J(Q P )/2J(Q P ) of (L*/L* 2 ) jv=i, 
then there is an ideal I of R with I 2 C aR and (NI) 2 = N(a). Hence there is an integral orbit 
representing the rational orbit of a. 

Proof: We prove this by an explicit construction of the ideal /. We can represent any class in 
J(Qp) /2J(Qp) by a Galois-invariant divisor of degree zero 

D = (A) + (P 2 ) + ■ ■ • + (P m ) - m(0) 

with m < n, where the Pj = (a^, 6j) are defined over some finite Galois extension K of Q p . We may 
assume that the coordinates of all the p lie in the integers of K with bi ^ 0, and that the are all 
distinct. Define the monic polynomial 

P(x) = (x — ai)(x — a 2 ) • • • (x — a m ) 

of degree m over Q p . By our explicit description of the coboundary map from J(Q P )/2J(Q P ) to 
H^Qp, J[2]), we have (-l) m P(/3) = a in (L^L* 2 )^. Next define R(x) as the unique polynomial 
of degree < m — 1 over Q p with the property that 

R(di) = bi, 1 < i < m. 

Then R(x) 2 — f(x) = m(x)P(x) in Q p [x]. This gives the Mumford representation of the divisor D, 
which is cut out on the curve by the two equations P(x) = and y = R(x). 

The polynomial P(x) clearly has coefficients in the ring Z p . When R(x) also has coefficients in 
Z p , we define the ideal Id of R = Z p [/3] as the P-submodule of L generated by the two elements P(/3) 
and R(/3). For example, when m = 1, the divisor D has the form (P) — (O) with P = (a, b), and the 
ideal I D is generated by the two elements (/3 — a) and b. 

We claim that I D is contained in the ideal generated by a = (— l) m P(f3) and that N(I D ) = 
b\b2 ■ ■ - bra- Hence N(I D ) 2 generates the same ideal as N(P((3)) = ( — l) m &^&| • • - b 2 m and we have 
constructed an integral orbit mapping to the rational orbit of a. 

The fact that the ideal I 2 D = (P(/3), P(/3)P(/3), R{(3) 2 ) is contained in the ideal generated by P{(3) 
follows from /(/?) = 0, so R(f3) 2 = m(/3)P{/3). On the other hand, 

R/I D = Z P [f3}/I D = Z p [x]/(f(x), P(x), R(x)) = Z p [x]/(P(x), R(x)) 

and so N(I D ) = #Z p [x]/(P(x), R(x)). The latter cardinality may be computed as the determinant 
(viewed as a power of p) of the Z p -linear transformation xi?(i) : Z p [x]/(P(x)) — > Z p [x}/ (P(x)). This 
is equivalent to computing the determinant of the K -linear transformation xR(x) : K[x]/(P(x)) — > 
K[x]/ (P(x)). But P(x) splits over K, and we have 

K[x]/(P(x)) ^ K[x]/(x - ai) x • • • x - a m ) ^ K m (12) 
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since the a; are distinct. The image of R(x) in K m above is given by (R(ai), . . . , R(a m )) = (b±, ... , b m ), 
and thus the determinant of xR(x) as a transformation of K[x]/ (P{x)) = K m is simply h\ • ■ ■ b m , as 
claimed. This completes the construction of an integral orbit when the polynomial R(x) has coeffi- 
cients in Z p . 

If R(x) is not integral, we consider the 2n + l roots of the monic polynomial F(x) — R(x) 2 . These 
contain the roots a\, a 2 , ■ ■ ■ , a m of P(x), and a consideration of the Newton polygon shows that at most 
m — 2 of the remaining roots r 1; r 2 , . . . , r m _ 2 have non-negative valuation. Let Sj = R(ri). Then the 
function ?/ — on the curve has principal divisor 

{(ai, 61) + .. + (a m , 6 m ) - m(O)} + {(n, si) + • • • + (r m _ 2 , s m _ 2 ) - (m - 2)(0)} + ,5 

where the x-coordinates of the points in the divisor E are the remaining roots. Since these all have 
negative valuation, the class of E lies in 2J(Q p ). Hence the class of the divisor D is equivalent to the 
class of the divisor 

D* = (n, s x ) + ■ ■ ■ + (r m _ 2 , s m _ 2 ) - (m - 2)(0) 

in the quotient group J(Q P )/2J(Q P ). The proof now concludes by a descent on the number m of 
points in D. Once m < 1 the polynomials P(x) and are both integral. □ 

We end with some brief remarks on integral orbits (J, a) in the case when Endfl(J) = R! is strictly 
larger than R, so / is not a free i?-module. When the order R! is Gorenstein, I is a free _R'-module, so 
can be identified with R'. Moreover, the dualizing module Hom(i?', Z p ) is free, and can be identified 
with (I/7) under the trace pairing, with 7 in L* chosen so that ^(7) = N(f'(f3)) in Q*/Q* 2 . 
The element 7 is well-defined up to multiplication by R'* 2 , and defines an integral orbit via taking 
the equivalence class of the pair (I, a) = (R',j/f'(f3)). So the integral orbits with characteristic 
polynomial f(x) and endomorphisms by a Gorenstein order R' containing R in L form a principal 
homogeneous space for the group (R 1 * / R'* 2 )n=i- They map to the rational orbits with classes 7//'(/3) 
in (L* / L* 2 ) N =i, where (I/7) generates the dualizing module of R! in L, and these rational classes 
need not lie in the unit subgroup of (L* / L* 2 ) n=i- 

8.2 General integral orbits 

The lattice W <g> Z 2 has rank 2n + 1 and discriminant disc(W) = (-1)" det(W) = 1 in Z 2 /Zf . 
In this case we need a bit more to identify those lattices isometric to W over Z 2 , as both orthogonal 
spaces over Q 2 of dimension 2n + 1 and discriminant = 1 contain non-degenerate lattices with these 
invariants. 

Lemma 19 Let I be a ^-module of rank 2n + 1 equipped with a symmetric bilinear form I x / — > Z 2 
whose discriminant is the square of a unit. Then I is isometric to W over Z 2 if and only if it contains 
an isotropic sublattice of rank n. 

Indeed, the existence of such an isotropic sublattice is equivalent to the assumption that / ® Q 2 is 
a split orthogonal space, and in this space all non-degenerate odd lattices are conjugate. The stabilizer 
of such a lattice is the normalizer of a parahoric subgroup of SO(W)(Q 2 ). 

If the bilinear lattice I contains a maximal isotropic sublattice over Z 2 , then we say that it is 
split [35]. The same argument as the proof of Proposition 15 yields the following. 
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Proposition 20 Assume that f(x) = x 2n+1 + c 2 x 2n ~ 1 + c 3 x 2?1 ~ 2 + • • • + c 2n +i is a polynomial with 
coefficients in Z 2 and non-zero discriminant in Q 2 . Then the integral orbits of 'G(Z 2 ) on self-adjoint 
operators T with characteristic polynomial f(x) correspond to the equivalence classes of pairs (I, a) 
for the order R = Z 2 [x]/(/(x)), with the property that the bilinear form ( , ) Q -i on I is split. The 
integral orbit corresponding to the pair (I, a) maps to the rational orbit of SO (W)(Q 2 ) corresponding 
to the class of a in (L*/L* 2 ) N = h 

The condition that the pair (/, a) gives a split orthogonal space is not empty. Consider the case 
when f(x) has integral coefficients and the discriminant of f(x) is a unit in 7h\. Then the ring R — Al 
is the maximal order in L, and every pair (J, a) is equivalent to (R, u) with u in the unit subgroup 
{A* L / A* 2 ) N =i of (L*/L* 2 ) N =i . This subgroup of has order 2 m+2n , where f(x) has m + 1 irreducible 
factors over Q 2 . Not all of these orthogonal spaces can be split over Q 2 , as a maximal isotropic 
subgroup for the quadratic form 77 on (L*/£* 2 )tv=i has order 2 m+n . On the other hand, the image of 
classes in J(Q 2 )/2 J(Q 2 ) give a subgroup of the units of order 2 m+n , which do correspond to orbits. 

We suspect that this remains true in general. More precisely, we conjecture that whenever f(x) has 
coefficients in Z 2 and a is a class in J(Q 2 )/2J(Q 2 ), the corresponding orbit over Q 2 is represented 
by (at least one) integral orbit over Z 2 . One can try to imitate the construction of an ideal that we 
gave in the case where p is odd. The Mumford representative works just as well, but the reduction to a 
divisor of smaller degree used the fact that when P = (a, b) on C has ord(a) < 0, the class (P) — (O) 
is divisible by 2 in the Jacobian. For p = 2 this is true if we assume that the integral coefficients 
satisfy: 2 4fc divides c^, for k — 2, 3, . . . , 2n + 1. So for integral polynomials of this form, each class in 
^(Q2)/2J(Q 2 ) is represented by at least one integral orbit. 

Similarly, we have the following result on integral orbits over Z. 

Proposition 21 Assume that f(x) = x 2n+1 + c 2 x 2n_1 + c 3 x 2n ~ 2 + • • • + c 2n+ i is a polynomial with 
coefficients in Z and non-zero discriminant in Q. Then the integral orbits of G(Z) on self-adjoint 
operators T with characteristic polynomial f(x) correspond to the equivalence classes of pairs (J, a) 
for the order R = Z[x]/(/(x)), with the property that the bilinear form ( , ) Q -i on I is split. The 
integral orbit corresponding to the pair (I, a) maps to the rational orbit ofSO(W)(Q) corresponding 
to the class of a in (L*/L* 2 ) N =i. 

Here one can check if the bilinear form is split simply by a calculation of the signature of the 
orthogonal space over IR (see [45, Chapter V]). 

In general, given a rational orbit with integral characteristic polynomial f(x), we can ask if it is 
represented by an integral self-adjoint operator T : W — > W. That is, given an element a in L* that 
represents a class in the group (L* / L* 2 ) n=i lying in the kernel of r], we can ask if there an ideal / of 
R = Z[x]/(f(x)) with I 2 C aR and (TV/) 2 = N(a). In this case, there is no need to check that the 
bilinear form defined by a on / is split, as it is split over Q. The number of integral orbits mapping to 
a fixed rational orbit is finite, as there are only finitely many ideals with a given norm. 

Fix a representative of the class a in the group L*, and let m(a) be the number of ideals I of R with 
the property that I 2 C aR and (NI) 2 = N(a). Let m p (a) be the number of ideals I p of R p := i? Cg> Z p 
with this property. For all primes p that do not divide A and where the element a is a unit, we must 
have I p = R p . Hence m p (a p ) = 1 for almost all p, where a p denotes the image of a in R p . Since an 
ideal of R is determined by its localizations, and since the intersection of L with the product JT I p is 
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an ideal I of R with the desired properties, we have the product formula 

m(a) = Y\_ m p( a )- 
v 

The pair (I, a) gives the same orbit as the pair (cJ, a) whenever c is an element of L* with c 2 = 1 and 
N(c) = 1. On the other hand, the ideals / and cl are equal only when c is a unit in the endomorphism 
ring R(I) of the ideal /. So the number of distinct ideals giving the same integral orbit is the order of 
the finite quotient group L*[2]jv=i/-R(/)*[2]jv=i- Note that the stabilizer of an operator in this integral 
orbit is the finite group R(I)*[2] N=1 and the stabilizer of an operator in the rational orbit a is the finite 
group L*[2]jv=i. Hence we also have the formula 

m (a) = ^#Stab(a)/#Stab(0) 
o 

where the sum is taken over the integral orbits mapping to the rational orbit. 

The same argument comparing ideals and integral orbits works locally, so combining the above 
formulas gives the following 

Proposition 22 Assume that the number of integral orbits representing a fixed rational orbit a is non- 
zero. Then 

#Stab(a) l/#Stab(0) = JJ #Stab(a p ) ^ l/#Stab(O p ) 
o P o P 

where almost all of the terms in the product are equal to 1. 

Finally, we have the following result on locally trivial orbits. 

Proposition 23 Let f(x) have coefficients in 7L, and assume that 2 4k divides c^for all k. Then every 
class in the 2-Selmer group of J is represented by at least one integral orbit. 

This follows from the construction of an ideal I p at each prime. The divisibility hypothesis was 
needed to construct I2 and an integral orbit over Z 2 . It is not necessary when the order R is maximal 
at 2, and we suspect that it is not necessary in general. 

9 Construction of fundamental domains 

In the previous sections, we have studied the rational and integral orbits of the representation of SO(W) 
on V, and shown how the locally soluble orbits are naturally related to the 2-Selmer groups of the 
Jacobians of hyperelliptic curves. In order to prove Theorem 1, we now turn to the question of counting 
these orbits. 

In this section, we construct convenient fundamental domains for the action of G(Z) on V(R), 
where G = G m x SO(W) and an element A G G m acts on V by scalar multiplication. In Section 9, 
we will then count the number of "irreducible" points in V(Z) in these fundamental domains having 
bounded height, which will then allow us to prove Theorem 1 in Sections 10 and 1 1 via the appropriate 
sieve methods. 
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We say that an element in V(Z) is reducible if it either lies in a distinguished orbit over Q or has 
discriminant zero; we say that it is irreducible otherwise. The height of an element B G V(M) and 
of its associated characteristic polynomial f(x) is defined as follows: if B G V(R) has associated 
invariants c 2 , . . . , C2 n +i given by 

/(z) = det(Ax - 5) = x 2n+1 + c 2 x 2n ~ 1 + ■■■ + c 2n+1 , 

then the height H(B) of 5 and the height H(f) of / is defined by 

H(B):=H(f) :=max{cf^/ fe |}S 1 . 

To describe our fundamental domains explicitly, we put natural coordinates on V. Recall that V 
may be identified with the n(2n + 3) -dimensional space of symmetric matrices having anti-trace zero. 
We use bij to denote the entry of B G V; thus the 6^ ((i, j) 7^ (n + 1, n + 1)) form a natural set 
of coordinates on V. 



9.1 Fundamental sets for the action of G(R) on V(K) 

Recall that we have naturally partitioned the set of elements in V(R) with A ^ into Ylm=o ( m+ ) 
components, which we denote by V^ m,T ^ for m = 0, . . . , n and r = 1, . . . , ( 2r ^ 1 ) • For a given value 
of m, the component y( m > r ) in V(R) maps to the component I{m) of characteristic polynomials in 
R 2n having nonzero discriminant and 2m + 1 real roots. 

We first construct fundamental sets L^ m ^ for the action of G(R) on {B G V (m ' r) : H(B) = 1} for 
each m, r. To this end, we fix a self-adjoint operator T : W ^ W in the distinguished orbit having a 
given characteristic polynomial /(x) with 2n + 1 real roots. The polynomial f(x) of T is separable, 
of trace and split over R. Let j*i < r 2 < ■ ■ • < r 2n +i be the roots of f(x) and let L = R[T] be 
the algebra centralizing T in End(VF). All of the operators in L are self-adjoint and have the same 
eigenvectors in W. We thus obtain an isomorphism L — > IR 2n+1 which maps the polynomial p(T) to 
the eigenvalues (x u . . . , x 2n+ i) = (p(ri), . . . ,p(r 2n+1 )). 

With these coordinates, consider the cone C defined by the inequalities x 1 < x 2 < ■ ■ ■ < x 2n +i 
and J2 x i = m me subalgebra of elements of trace in L. The operators in this cone represent 
the distinguished orbits with split characteristic polynomial, as the eigenvectors for the eigenvalues 
X\ < x 2 < ■ ■ ■ < x 2n+ i are the same as the eigenvectors for T. If we scale the elements in C to have 
height 1 we are in a bounded region. This is the set Z^™' 1 ) of G(R) -representatives of height 1 that we 
take for V^ l \ 

Now take another V^ n ' T ^ where the characteristic polynomial splits: r is given by a collection of 
sign changes a in L* = IR* 2n+1 . If T' in the region has characteristic polynomial f(x), then we can 
find an element g G GL(V)(R) with gTg" 1 = T' and g*g = a G L*. The elements g(vi) are now 
eigenvectors for T' with the appropriately changed signs. 

For any S in our compact region L^ 1 ^, the operator gSg^ 1 = S' is self-adjoint and lies in the new 
region L^ m ' T ^ = gL^'^g^ 1 . The operator S' possesses the same characteristic polynomial as S, so we 
obtain a fundamental set L^ m ^ of G(M) -representatives for ^/( m > T ) which also lies in a compact subset 
(as conjugation by a fixed g is continuous). 

Thus we have chosen our fundamental sets L^ 71 ^ so that the entries of all elements B in L( m > T ) 
having height less than 1 are bounded; i.e., the L^ m ' T ^ all lie in a compact subset of V(R). Note that for 
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any fixed h G G(M) (or for any h lying in a fixed compact subset Go C G(R)), the set hL^ m,T ^ is also 
a fundamental set for the action of G(M) on V^ m ' T \ and again all coefficients are bounded. 



9.2 Fundamental sets for the action of G(Z) on G(R) 



In [8], Borel (building on earlier work of Borel-Harish-Chandra [9]) constructs a natural fundamental 
set T in G(M) for the left action of G(Z) on G(R). This set T may be expressed in the form 

T = {utOX : u G N'(t), teT',9 G AT, AG A}, 

where iV'(t) is an absolutely bounded measurable set, which depends on t G A', of unipotent upper 
triangular real (2n + 1) x (2n + 1) matrices; the set T' is the subset of the torus of diagonal matrices 
with positive entries given by 



r 



\ 



tn 1 



I V 



: ti/t 2 > c, t 2 /t 3 > c, . . . , t n -i/t n > c, t n > c > 



(13) 

K is a maximal compact real subgroup of G(R); and A = {A > 0} acts on V(M) by scaling. In the 
above, c denotes an absolute positive constant. (Note that U/t i+ i for i = 1, . . . , n — 1 and t„ form a set 
of simple roots for our choice of maximal torus T in G ss = SO(W) C G.) 

It will be convenient in the sequel to parametrize T' in a slightly different way. Namely, for 
k = 1, . . . , n, we make the substitution t fc = SkSk+i ■ ■ ■ s n . Then we may speak of elements s = 
(si, . . . , s n ) G T' as well, and the conditions on s for s — (si, . . . , s n ) to be in T' is simply that > c 
for all k = 1 n. 



9.3 Fundamental sets for the action of G(Z) on V(R) 

For any /i G by the previous section we see that FhL^™ 1 ^ is the union of 2 m+n fundamental 

domains for the action of G(Z) on ^/( m > T ); here, we regard J 7 hL^ m ' T ' as a multiset, where the mul- 
tiplicity of a point x in ThlP n ' ,T > is given by the cardinality of the set {g G T : x G ghL^ m ' T ^}. 
Thus a G(Z) -equivalence class x in y( m > r ) is represented in this multiset a(x) times, where a{x) = 
#StabG( K )(x)/#StabG(z)(x). In particular, a(x) is always a number between 1 and 2 m+n . 

For any G(Z) -invariant set S C V im ' r) nV (Z), let iV(5; X) denote the number of G(Z) -equivalence 
classes of irreducible elements B G S satisfying H(B) < X. Then we conclude that, for any 
h G C(R), the product 2 m+n ■ N(S; X) is exactly equal to the number of irreducible integer points in 
ThL^ m ' T > having height less than X, with the slight caveat that the (relatively rare — see Lemma 31) 
points with G(Z)-stabilizers of cardinality r (r > 1) are counted with weight \/r. 

As mentioned earlier, the main obstacle to counting integer points of bounded height in a single 
domain J 7 hL^ m ' T ' is that the relevant region is not compact, but rather has cusps going off to infinity. 
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We simplify the counting in this cuspidal region by "thickening" the cusp; more precisely, we compute 
the number of integer points of bounded height in the region ThU m,T ^ by averaging over lots of such 
fundamental regions, i.e., by averaging over the domains J r hL^ m ' T ^ where h ranges over a certain 
compact subset Go G G(R). The method, which is an extension of the method of [1], is described next. 



10 Counting irreducible integral points of bounded height 

In this section, we derive asymptotics for the number of G(Z)-equivalence classes of irreducible el- 
ements of V(Z) having bounded invariants. We also describe how these asymptotics change when 
we restrict to counting only those elements in V(Z) that satisfy any specified finite set of congruence 
conditions. 

Let V (m ' T) (m = 1, . . . , n, r = 1, . . . , ( 2 ™ +1 )) denote the orbits of G(R) on the subset of V(R) 
having nonzero discriminant, and let 

Vol(.TL( m ' T ) n{i)6 V(R) : H{y) < 1}) 

^m,T 2m+n 

Then in this section we prove the following theorem: 

Theorem 24 Fixm,r. For any G(Z) -invariant set S C ViZ)^^ := V z nV^ T \ letN{S;X) denote 
the number of G(Z) -equivalence classes of irreducible elements B G S satisfying H(B) < X. Then 

N{V{Z)^ T) -X) = c miT X( 2n+3 )/( 4n+2 ) + (x( 2n+3 )/( 4n+2 )). 



10.1 Averaging over fundamental domains 

Let Go be a compact left A'-invariant set in G(M) that is the closure of a nonempty open set and in 
which every element has determinant greater than or equal to 1. Let V(Zy rT denote the subset of 
elements of V(Z) that are irreducible. Then for any m, r, we may write 

f #{ x G ThL n V(Zf r : H(x) < X}dh 
N(V(Z)^; X) = ^Go^ __ ^ 1_ r_ (14) 

where V(Z) 1TT denotes the set of irreducible elements in V(Z) 1TT and L is equal to L^ m ' T \ The denomi- 
nator of the latter expression is an absolute constant C^'^ greater than zero. 

More generally, for any G(Z)-invariant subset S C V{Z) (jn,T \ let N(S; X) denote the number of 
irreducible G(Z) -orbits in S having height less than X. Let S WT denote the subset of irreducible points 
of S. Then N(S; X) can be similarly expressed as 

f #{ x G ThL n S iTT : H(x) < X}dh 
N(S; X) = K -L (15) 

We use (15) to define N(S; X) even for sets S C V(Z) that are not necessarily G{Z) -invariant. 



30 



Now, given x G V^ n ' T \ let xl denote the unique point in L that is G(R) -equivalent to x. We have 

= _I_ [ #{g^?:x = ghx L }dh. (16) 

C Go xeS ^ JheGo 
H(x)<X 

For a given x G S nr , there exist a finite number of elements g 1} . . . , g k G G(M) satisfying gjXL = x. 
We then have 



/ #{g EJ r :x = ghx L }dh = / #{# G T : gh = g^dh = V / d/z, 

AeGo j JheGo j JheGofXF^gj 



As dh is an invariant measure on G, we have 



V" / dh = y^ dg = y^ ^G Q : gh = g^dg = / eG :i = ghx L }dg. 

j AeGoOF- 1 ^ j JgeGog^nT- 1 j J g^T Jg&T 

Therefore, 



(17) 



(18) 



N{S-X) = — L> [ #{heG :x = ghx L }dg. 

H(x)<X 

= I E Si " n 9GoL : H(x) < X} dg 

C G ' J 9& 

= -iy I #{x G S in ~ n us\9G L : H{x) < X}dg, (19) 

C G ' Jg£N'{s)A'KK 

where dg is a Haar measure on G(R). Explicitly, we have 

n n 

dg = JJtf" 2n_1 ■ dud x td x \d6 = Y[ sf- 2kn ■dud x td x \d0, 



k=l m=l 



where du is an invariant measure on the group N of unipotent upper triangular matrices in Gu, and 
where we normalize the invariant measure d6 on K so that J K d9 — 1 . 

Let us write E(u, s, A, X) = us\G L n{x E : H(x) < X}. As A'G = G and / x dfl = 1, 

we have 

1 r n 

N(S-X) = —— / #{x E S hr nE(u,s,\,X)}T\st 2 ~ 2kn ■ dud x sd x \. (20) 

C Go ' JgeN>(.)A>A fe=1 

We note that the same counting method may be used even if we are interested in counting both 
reducible and irreducible orbits in V(Z). For any set S C V^ m ' T \ let N*(S; X) be defined by (20), but 
where the superscript "irr" is removed: 

1 r n 

N*(S;X) = ^^ / #{xeSnE(u,s,\,X)}T[sl 2 - 2kn -dud x sd x \. (21) 
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Thus for a G z -invariant set S C V^ m ' T \ N*(S; X) counts the total (weighted) number of G(Z)-orbits 
in S having height less than X (not just the irreducible ones). 

The expression (20) for N(S; X), and its analogue (21) for N*(S, X), will be useful to us in the 
sections that follow. 

10.2 An estimate from the geometry of numbers 

To estimate the number of lattice points in E(u, s, A, X), we have the following proposition due to 
Davenport [19]. 

Proposition 25 Let 1Z be a bounded, semi-algebraic multiset in W 1 having maximum multiplicity m, 
and that is defined by at most k polynomial inequalities each having degree at most t. Let TZ' denote the 
image oflZ under any (upper or lower) triangular, unipotent transformation ofW L . Then the number 
of integer lattice points (counted with multiplicity) contained in the region TZ' is 

Yo\(7Z) + 0(max{Vol(ft), 1}), 

where Xo\(TZ) denotes the greatest d-dimensional volume of any projection of TZ onto a coordinate 
subspace obtained by equating n — d coordinates to zero, where d takes all values from I ton — I. The 
implied constant in the second summand depends only on n, m, k, and i. 

Although Davenport states the above lemma only for compact semi-algebraic sets TZ C W 1 , his 
proof adapts without significant change to the more general case of a bounded semi-algebraic mul- 
tiset TZ C M. n , with the same estimate applying also to any image 1Z' of TZ under a unipotent triangular 
transformation. 

10.3 Estimates on reducibility 

In this subsection, we describe the relative frequencies with which reducible and irreducible elements 
sit inside various parts of the fundamental domain ThL, as h varies over the compact region G - 

We begin by describing some sufficient conditions that guarantee that a point in V(Z) is reducible. 
Recall that a point B G V(Z) is called reducible if it lies in a distinguished rational orbit or it has 
discriminant zero. 

Lemma 26 Let B G V (Z) be an element such that, for some k G {1, . . . , n}, the top leftkx (2n+l— k) 
block of entries of B are all equal to 0. Then Disc(Det(Ac — B)) = 0, and thus B is reducible. 

Proof: If the top left k x (2n + 1 — k) block of B is zero, then B (and indeed Ax — B for any x £ C) 
may be viewed as a block anti-triangular matrix, with square blocks of length k, 2n + 1 — 2k, and k 
respectively on the anti-di agonal. Furthermore, the two blocks of length k will be transposes of each 
other. There then must be a multiple Axq of A such that Axq + B has the property that one and thus 
both of these square blocks of length k are singular. It follows that x is a double root of Det(Ax — B), 
yielding Disc(Det(Ac — B)) = 0, as desired. □ 

We also have the following consequence of Proposition 9 and Lemma 26: 
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Lemma 27 Let B = (b^) G V(Z) be an element such that, for all pairs satisfying i+j < 2n + l, 
we have = 0. Then B is reducible. 

We are now ready to give an estimate on the number of irreducible elements B = (bij) G ThL D 
V(Z), on average, satisfying b\\ = 0: 

Proposition 28 Let h take a random value in Go uniformly with respect to the Haar measure dg. Then 
the expected number of irreducible elements B G J r hL ( - m ' T ^ n V(Z) such that H(B) < X and bn = 

2n+3 1 
isO £ {X in + 2 2n(2n+l) +£ ). 

Proof: Let U denote the set of all n{2n + 3) variables b^ corresponding to the coordinates on V(Z). 
Each variable b G U has a weight, defined as follows. The action of s = (si, . . . , s n ) ■ A on B G 
K causes each variable 6 to multiply by a certain weight which we denote by w(b). These weights 
w(b) are evidently rational functions in A, si, . . . , s n . If we use w±, . . . , u>2„ to denote the quantities 
si, S2, • • • , s n , s n ,..., s 2 , si, respectively, then we have the explicit formula 

w(bij) = \ ■ s± 2 ■ ■ ■ s~ 2 ■ (wi ■ ■ ■ Wi-i) ■ (wt ■ ■ ■ Wj-x). (22) 

In particular, if i + j = 2n + 2, then we see that w(bij) — A. If i + j = 2n + 1, with i < n, then 
wipij) = As" 1 . Finally, note that \~[ beU w{b) = A n ( 2n+3 ). 

Let Uq C U be any subset of variables in U containing b u . We now give an upper estimate on the 
total number of irreducible B G J r hL^ m ' T ' ) n V(Z) such that all variables in f/ vanish, but all variables 
i n [/ \ U do not. To this end, let V(U ) denote the set of all such B G V(Z). Furthermore, let U\ 
denote the the set of variables having the minimal weights w(b) among the variables b G U \U (by 
"minimal weight" in U \ U , we mean there is no other variable u G U \ T with weight having equal 
or smaller exponents for all parameters A, si, . . . , s n ). 

If Ui contains any variable b^ with i+j > 2n + 1, then any element B G V(Uq) will be reducible 
by Lemma 26. If U\ consists exactly of those variables b^ for which i + j = 2n + 1 (i.e., B contains 
all variables by with i + j < 2n + 1), then any element 5 G V(f/ ) will be reducible by Lemma 27. 
Thus, we may assume that all variables b^ G U\ satisfy % + j < 2n + 1, and at least one variable b^ 
satisfies i + j < 2n + 1. 

In particular, there exist variables . . . , f3 n G C/i such that, for all k, the weight w(/3 k ) has equal or 
smaller exponents for all parameters A, s%, . . . , s n when compared to the weight w(bk,2n+i~k) = As^ 1 . 
For each subset Uq C f/ as above, we wish to show that A/"(V(C/o); X), as defined by (20), is 

2n+3 _ 1 

O e (X 4n + 2 2n(2«+i) £ ) since A/"'(s) is absolutely bounded, the equality (21) implies that 

X l/(2n(2n+l)) 

iV*(y([/ );X)« / / a(V(f/ ))n s f" 2fcn - rfXsrf><A ' ( 23 ) 

J \=C' Js!,...,S n =C fc = 1 

where cr(^(?7o)) denotes the number of integer points in the region E(u, s, A, X) that also satisfy the 
conditions 

b = for b G C/ and |b| > 1 for 6 G Z7 X . (24) 
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By our construction of L, all entries of elements B £ GqL are uniformly bounded. Let C be a 
constant that bounds the absolute value of all variables b E U\ over all elements B £ G £- Then, for 
an element £> £ .E^m, s, A, X), we have 

|6| < Cw(b) (25) 

for all variables b £ U±', therefore, the number of integer points in E(u, s, A, X) satisfying 6 = for all 
b £ Uq and |6| > 1 for all b £ U\ will be nonzero only if we have 

Cw{b) > 1 (26) 

for all weights w(b) such that b £ U±. Thus if the condition (26) holds for all weights w(b) correspond- 
ing to b £ U±, then — by the definition of U\ — we will also have Cw(b) 3> 1 for all weights w(b) such 
that b £ U \ Uq. In particular, note that we have 

Cw(p k ) > 1 (27) 

for all k — 1, . . . , n. 

Therefore, if the region B = {B £ E(u, s, A, X) : 6 = V6 £ C/ ; |6| > 1 V6 e *7i} contains 
an integer point, then (26) and Lemma 25 together imply that the number of integer points in B is 
0(Vo\(B)), since the volumes of all the projections of u~ x B will in that case also be 0(Vo\(B)). Now 
clearly 

Vol(B) = o( J] «;(&)), 
6ec/\a 

so we obtain 

X l/(2n(2n + l)) „ 

N(y(U );X)<^ / J] w (6) J] S f- 2fcn -rf x srf x A. (28) 

JX=c' Jsi,...,s n =c beU ^ Uo fc=1 

2n+3 _ 1 +& 

We wish to show that the latter integral is bounded by £ (X 4n+2 2n(2n+i) j f or ever y choice of 
C7o< If. f° r example, the total exponent of Sk in (28) is nonpositive for all k in {1, ... , n}, then it is 
clear that the resulting integral will be at most O e (X^ 2n+3) ~^ Uo ^^ 2n( - 2n+l ^ +£ ) in value, since each s k 
is bounded above by a power of X by (27). 

For cases where this nonpositive exponent condition does not hold, we observe that, due to (27), the 
integrand in (28) may be multiplied by any product tt of the variables 0k (k — 1, . . . , n) without harm, 
and the estimate (28) will remain true. Extend the notation w multiplicatively, i.e., w(ab) = w(a)w(b), 
and for any subset U' C U, write w([\ beU , b) = A'^ 7 ' • Yl • Then we set 



n = n(U ) =l[(3 



max{0 ,e k (U )+k 2 -2kn} 
k 

k=l 



We may then apply the inequalities (25) to each of the variables in tt, yielding 

„ x l/(2n(2n+l)) n 



N(y(U );X)4Z / w(b)w(7r)l[sf- 2kn -d x sd x X. (29) 

J A=c' J si,... ,s n =c ,^tt\ TT. u—i 



i,...,s n -c b£U \ Uo k= i 
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By the definition of /3 k , we have w((3k) \s k , implying 

jfl/(2n(2n+l)) n 

N(V(U );X)<^ / / w(b) A w Y[sf- 2kn - m3x{0 ' ek{Uo)+k2 - 2kn} -d x sd x \. 

J\=c' Ja 1 ,...,s n =c beU \ Uo k =l 

(30) 

In the above, we have chosen the factor n so that the total exponent of each s m in (30) is clearly 
negative. Thus we obtain from (30) that N(V(U ); X) = £ (X^ 2n+ ^-* Uo+ *^/^ 2n+1 ^ +£ ), where 
#7r denotes the total number of variables of U appearing in n (counted with multiplicity). It thus 
suffices now to check that we always have #£/ — #n(Uo) > 0. 

Let U~ denote the subset of variables bij in U such that i + j < 2n + 1. Then it is clear by the 
definition of % that replacing U by U D U~ can only decrease the value of #C/ — #ir(U ). It thus 
suffices to check the condition #C/ — #ir(U ) > for proper nonempty subsets U of U~, and this is 
the content of the following combinatorial lemma. 

Lemma 29 Let U Q U~. Then 

n 

^max{0, e k (U ) + k 2 - 2kn} < \U \ (31) 

k=l 

with equality if and only ifUo = <f) or Uq = U~~ . 

Proof: We prove the lemma by induction on n. The lemma is directly verified for n = 1. If n > 1, 
then we note that the weights for Sk (k > 1) agree with the weights for Sk-i when n is replaced by 
n — 1. Let U (l) (resp. U (> 2)) denote the set of all variables by E U such that i = 1 (resp. i > 2), 
and define U~(l) and U~(> 2) analogously. Then, by the induction hypothesis, we have 

ft 

^max{0 , e k (U ) + (k - l) 2 - 2(k - \){n - 1)} < \U (> 2)| (32) 

k=2 

with equality if and only if U (> 2) = or U (> 2) = U~(> 2). 

Let |C/o(l)| = r, so that r e {0, . . . , 2n — 1}. Then to prove (31), we may assume that U (l) = 
{bn, bi2, . . • , bir}, for any other choice of U (l) C t/ _ (l) of size r could only decrease the sum on the 
left hand side of (31). 

We first consider the case where r < n; in that case, we have 

n 

max{0 , ei(C/ (l)) + 1 - 2ra} + max{0 , e fc (t7 ) + k 2 - 2kn} 

k=2 

n 

= + max {° > e k (U (> 2)) + (A; - l) 2 - 2{k - l)(n - 1) + e fc (C/ (l)) + (1 - 2n)} 

fc=2 

n 

< |C/ (> 2)| + ^ max{0 , e k (U (l)) + (1 - 2n)} 

k=2 

= \U {>2)\ < |E/o|, 
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with equality if and only if r is zero and all terms on the left hand sides of (31) and (32) are zero, i.e., 
Now consider the case where r > n. In that case, we have similarly 

n 

max{0 , e k (U ) + k 2 - 2kn} 

k=l 

n 

< \U Q (>2)\ + J2 max {° > e fc (C^ (l)) + (1 - 2n)} 

k=l 

= |Z7 (>2)| + 2(r-7i-l)-l<|Z7o|, 

with equality if and only if r is 2n — 1 and we have equality in (32) with all terms nonzero, i.e., 

U Q = U~. □ 

This completes the proof of Proposition 28. □ 

We now give an estimate on the number of reducible elements B = (b^) E ThL fl V(Z), on 
average, satisfying b n ^ 0: 

Proposition 30 Let h take a random value in Gq uniformly with respect to the measure dg. Then the 
expected number of reducible elements B E J r hL^ m,T ^ fl V(1*) such that H(B) < X and b u ^ is 

p^(2n+3)/(4n+2)y 

We defer the proof of this lemma to the end of the section. 

We also have the following proposition which bounds the number of G(Z) -equivalence classes of 
integral elements in ThL^ m,T ' with height less than X that have large stabilizers inside G(Q); we again 
defer the proof to the end of the section. 

Proposition 31 Let h £ Go be any element, where Gq is any fixed compact subset ofG(M.). Then the 
number of integral elements B e Tlx{hL^ n ' T ^) whose stabilizer in G(Q) has size greater than 1 is 

^ x (2n+3)/(4n+2)y 

10.4 The main term 

Fix again m, t and let L = L^ m ' T l The work of the previous subsection shows that, in order to obtain 
Theorem 24, it suffices to count those integral elements B E ThL of bounded height for which bn 7^ 0, 
as h ranges over G . 

Let IZx(hL) denote the region FhL fl {B E Vr : H(B) < X}. Then we have the following result 
counting the number of integral points in 1Z x (hL), on average, satisfying bn ^ 0: 

Proposition 32 Let h take a random value in Go uniformly with respect to the Haar measure dg. Then 
the expected number of elements B E ThL fl V(ZY m ' T ^ such that \H(B)\ < X and bn ^ is 

2n+3 _ 1 

Vo\(K x (L)) + 0(X 4 «+ 2 2 "(2n+l)). 
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Proof: Following the proof of Lemma 28, let V^ m ' T \(f)) denote the subset of Vr such that bn ^ 0. We 
wish to show that 

2n+3 _ 1 

N*(V {m ' T \<p);X) = Vol(Kx(L)) + 0(X in + 2 M2n+i)). (33) 

We have 

1 „x 1 /( 2 "( 2 "+ 1 » poo p n 

N\V^\<P)-X) = —— / / a(V(<f>))l[sf-^-dud*sd*\, (34) 

C^ o ' ' J\=d J Sl ,...,s n =cJueN'(s) k=1 

where a(V((p)) denotes the number of integer points in the region B(u, s, A, X) satisfying |6 n | > 1. 
Evidently, the number of integer points in B(u, s, A, X) with \b u \ > 1 can be nonzero only if we have 

Cw(b u ) = C ■ — ->1. (35) 

S 1 S 2 ' ' ' S n 

Therefore, if the region B = {B G B(u, s, \, X) : \b u \ > 1} contains an integer point, then (35) 
and Lemma 25 imply that the number of integer points in B is Vol(i3) + 0(Vol(£>) /w(bn)), since all 
smaller-dimensional projections of u~ x B are clearly bounded by a constant times the projection of B 
onto the hyperplane b\\ = (since bn has minimal weight). 

Therefore, since B = B(u, s, A, X) — (B(u, s, A, X) — B) , we may write 



N *(y^\4>)-X) = -^rf ^ I (Vo\(B(u,s,X,X))-Yo\(B(u,s,X,X)-B) 

Cq q ' 'J\=c> J Sl ,...,s n =cJu£N'( S y- 

n 

+0(max{A™ (2n+3) s^ ■ ■ ■ s 2 n , 1})) JJ sf ~ 2kn -dud x s d x X. (36) 



k=l 



The integral of the first term in (36) is f heG Vo\(lZx(hL))dg. Since Vo\(lZx(hL)) does not depend 

on the choice of h G Go, the latter integral is simply Cq£ t ' ■ Vo\(TZx{L)). 

To estimate the integral of the second term in (36), let B' = B(u, s, t, X)—B, and for each \bn | < L 
let B'(bn) be the subset of all elements B G B' with the given value of bn. Then the (n(2n + 3) — 1)- 



dimensional volume of B'(bn) is at most O {Y[ beU \{ bll } w ) )j > an( ^ so we nave me estimate 

Vol(£')< / JJ w(b) db ll = o( K \\ w(b)y 
J' 1 beu\{b n } beu\{b u } 

The second term of the integrand in (36) can thus be absorbed into the third term. 

2n+3 1 



Finally, one easily computes the integral of the third term in (36) to be 0(X 4n+2 2n ( 2n + 1 ) ). We 
thus obtain 

2n±3_ 1 

N*(V {m ' T) ;X) = Xo\(n x (L)) + 0(X in + 2 Mw)^ ( 37 ) 

as desired. □ 

Propositions 28, 30, 31, and 32 now yield Theorem 24. 
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10.5 Computation of the volume 

Fix again m, t. In this subsection, we describe how to compute the volume of TZx{L^ m,T ^). 

To this end, let := AL^ and let G ss := SO(W). For each (c 2 , . . . , c 2n+1 ) G R 2n , the set 

i?( m ' T ) contains at most one point p( m ' T \c 2 , . . . , c 2n +i) having invariants c 2 , . . . , c 2n +i- Let R^ m ' T \X) 
denote the set of all those points in R( m > T ) having height less than X. Then Vol(TZx(L^ m ' T ^)) = 
VolOP 8 ■ R^ T \X)), where F ss denotes the fundamental domain N'A'K for the action of G SS (Z) on 
G SS (R) (here N', A', and K are as in (13)). 

The set R^ is in canonical one-to-one correspondence with the set {(c 2 , . . . , c 2n+ \) G M 2n : 
A(c 2 , . . . , c 2n+ i) > 0} if n - m is even, and with {(c 2 , . . . , c 2n+ i) G M 2n : A(c 2 , . . . , c 2n +i) < 0} if 
n — m is odd. There is thus a natural measure on each of these sets R^ m ' T \ given by dr = dc 2 ■ ■ ■ dc 2n+ i. 
We have the following proposition: 

Proposition 33 There exists a rational constant J such that, for any measurable function <fi on V(R), 
we have 

4>(g-p^ T \c 2 ,...,c 2n+1 ))dgdr= [ 4>(v)dv. (38) 



We know that G ss ■ R^ 1 ^ is an 2 m+n -fold cover of V ( m ' T ^ . The proposition then follows from a Jacobian 
computation and can be verified directly. 

There is an alternative method of arriving at (38) that does not require one to use any explicit 
formulas for the fundamental sets L^ m ' T >. We prove the following proposition, which will also be of 
use to us in the sequel. 

Proposition 34 Let dg be a choice of Haar measure on G(C). Let dv be the standard Euclidean 
measure on V(C). Let R be a subset ofV(C) that contains exactly one point p(c 2 , . . . , c 2n+ i) having 
invariants equal to c 2 , . . ., c 2n+ ifor each (c 2 , . . . , c 2n +i) G C 2 ™ and such that G R is measurable for 
any measurable set Go C G(C). 

Then there exists a constant J G Q such that, for any measurable function <fi on V(C), 

J_ 

2 2n 




<j)(g-p(c 2 ,...,c 2n +i))dgdr= / <j)(v)dv. 

RJG(C) Jv€V(C) 



Proof: We first prove that G(C) ■ R is a 2 2n -fold cover of a full measure set in V(C). To do 
this, we consider the set R consisting of the distinguished elements B(c 2 , . . . , c 2n+ i), as constructed 
in §4.1, for each (c 2 , . . . , c 2n+i ) G C 2n . Now if p(c 2 , . . . , c 2n+i ) G R has invariants such that 
A(p(c 2 , . . . ,c 2n+ i)) 7^ 0, then there exist 2 2n elements g(c 2 , . . . , c 2n +i) G G(C) such that g(c 2 , ■ ■ - ,c 2n+1 ] 
p(c 2 , . . . , c 2n+ i) = q(c 2 , . . . , c 2n+ i). The proposition can then be checked directly for R = R via a 
Jacobian computation, yielding the corresponding value of J. We may use this fact then to derive the 
result for general R as follows: 



4>(v)dv = J ■ / <t>{g- q(c 2 , • • • , c 2n+1 ))dgdr = J ■ / / (j)(gg(c 2 , . . . , c 2n+1 ) ■ p(c 2 , 

veG{C)-R JRoJG(C) JrJg(c) 



J- / <!>{g -P(c2, • • • ,c 2n+ x))dgdr, 

'-R JG(C) 
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where the last equality follows from the fact that G(C) is a unimodular group (see [29, Chapter 8]), 
i.e., the left Haar measure dg is also a right Haar measure on G(C). □ 

Proposition 34, along with the principle of permanence of identities, implies an analogous result 
when C is replaced with R or Q p for any prime p. In particular, it implies Proposition 33, which may 
now be used to give a convenient expression for the volume of the multiset TZx(L^ m ' T ^): 



[ dv= [ dv = \J\- [ [ 



dg dr 



= \J\ ■ Vol(G ss (Z)\G ss (M)) ■ / dr 

= \J\-Vol(G ss (Z)\G ss (R))- [ dc 2 ---dc 2n+1 , (39) 

J H(c 2 ,...,c 2n+1 )<X 
±A(c 2 ,...,c 2n+ i)>0 

where the sign ± depends on whether n — mis even or odd. 
10.6 Congruence conditions 

In this subsection, we prove the following version of Theorem 24 where we count elements of V(Z) 
satisfying some finite set of congruence conditions: 

Theorem 35 Suppose S is a subset ofV{Z) defined by congruence conditions modulo finitely many 
prime powers. Then 

N{S nV (rn ' T) ;X) = N{V{Z)^ T) ]X) ■ JJ// P (S) + (x( 2n+3 )/( 4n+2 )), (40) 

v 

where \x v {$) denotes the p-adic density of S in V(Z). 

To obtain Theorem 35, suppose S is defined by congruence conditions modulo some integer m. 
Then S may be viewed as the union of (say) k translates Ci,...,£k °f me lattice m ■ V(Z). For 
each such lattice translate Cj, we may use formula (20) and the discussion following that formula to 
compute N(S; X), but where each <i-dimensional volume is scaled by a factor of l/m d to reflect the 
fact that our new lattice has been scaled by a factor of m. For a fixed value of m, we thus obtain 

N{Cj n V {m ' T) - X) = m-^ 2n+ ^Vo\(R x (L)) + (x( 2 ™ +3 )/( 4 "+ 2 )). (41) 

Summing (41) over j, and noting that km~ n ^ 2n+3 ^ = Ylp^pi^), yields Theorem 35. 

We will also have occasion to use the following weighted version of Theorem 35; the proof is 
identical. 

Theorem 36 Let pi, . . . ,pk be distinct prime numbers. For j = 1, . . . , k, let 4> Pj : V(Z) — > ffi. be a 
G(Z) -invariant function on V(Z) such that <j> Pj (x) depends only on the congruence class of x modulo 
some power p^ 3 of pj. Let N ( f > {V(Z) D V^ m ' T ^;X) denote the number of irreducible G(Z)-orbits in 
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V(Z) D V( m ' T * > having height bounded by X, where each orbit G(Z) ■ B is counted with weight 4>{B) : = 

n-=i<Ms). Then 

N+(y(z) n v im ' r) ; x) = N(v(z) n k m ; x) TT / PJ (/) d/ + (x( 2ri+3 )/( 4n+2 )), (42) 



1 JfeV(i. Pj ) 

where (p Pj is the natural extension of <p Pj to V(Z Pj ) by continuity, df denotes the additive measure on 
V(Z Pj .) normalized so that Jf eV r z \df — 1, and where the implied constant in the error term depends 

only on the local weight functions 4> Pj . 

10.7 Proof of Propositions 30 and 31 

We may use the results of the previous subsection to prove Propositions 30 and 3 1 . Indeed, to prove 
Proposition 30, we note that if an element B £ V(Z) is reducible over Q then it also must be reducible 
modulo p for every p. 

Let 5' red denote the set of elements in V(Z) that are reducible over Q, and let Sp ed denote the set of 
all elements in V(Z) that are reducible mod p. Then S Tcd C n p S r p cd . Let S rcd (Y) = n p<Y S r p cd , and let 
us use as before V(0) to denote the set B £ V(Z) such that b u ^ 0. Then the proof of Theorem 35 
(without assuming Propositions 30 and 31 !) gives that 

N*{S rcd {Y) n V{4>)\ X) < 2 2n ■ N*(V(<f))] X) ] J fi p {S v p cd ) + (x( 2n+3 )/( 4n + 2 )). (43) 

p<Y 

To estimate fi p (Sp ed ), we recall from Subsection 6.1 that the number of elements B £ V(F p ) having 
any given associated separable polynomial f(x) = det(Ax — B) is #SO(W)(W p ). The number of 
these elements B that lie in the distinguished orbit is #SO(W^)(F p )/2 m , where m + 1 is the number of 
irreducible factors of / in ¥ p [x]. For p > 2n + 1, it is an elementary and well-known calculation that 
the number of separable polynomials 

/( X )= X 2n + l +C2X 2n-l + ... + C2n+i 

over F p that are reducible over F p is p 2n ■ 2n/(2n + 1) + 0(p 2n ~ l ), where the implied O-constant 
is independent of p. Since such a reducible polynomial f(x) has at least 2 factors in ¥ p [x], we see 
that at least #SO(VT)(F p )/2 of the elements B having associated polynomial f(x) do not lie in a 
distinguished orbit. We conclude that 

?n 1 

)£1 -2^T2 + ° (1/P) - 

Combining with (43), we see that 

N*(S^ d HV (0); X) _ „ , cred . tt / 1 



« n^ cd ) « n( i -^i+°( i /p))- 



jfe x( 2 "+3)/( 4 "+ 2 ) ^ 11 > ^ 11 ^ 2n + 

When F tends to infinity, the product on the right tends to 0, proving Proposition 30. 
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We may proceed similarly with Proposition 31. If an element B e V(Z) with nonzero discriminant 
has a nontrivial stabilizer, then the associated polynomial f(x) = det(Ax — B) must be reducible in 
Q[x], and thus must be reducible in ¥ p [x] for all p. 

Let S bisstab C V(Z) denote the elements 5 e y(Z) such that /(a;) = det(Ac - 5) is reducible in 
Q[x], and let S^igstab c ViZ) denote the elements B e V{Z) such that f(x) is reducible in ¥ p [x]. Let 
5 bigstab (F) = n p<yi S bigstab . Then we have by the same argument that 



p<y p<y 
Letting Y tend to infinity now proves also Proposition 31. 



11 Sieving to Selmer elements 

If F is a set of hyperelliptic curves (1) over Q defined by local congruence conditions, then we use F inv 
to denote the set {(c 2 (C), . . . , c 2n+ i(C)) : C 6 F}. We denote the p-adic closure of F inv in Z 2n by 
F' nv . We say that such a set F of hyperelliptic curves over Q is large at p if it contains all hyperelliptic 
curves C in (1) such that p 2 \ A(C). We say that a set F of hyperelliptic curves is large if it is large at 
all but finitely many primes p. In this section in conjunction with the work of Section 9, we will prove 
the following strengthening of Theorem 1 : 

Theorem 37 When all hyperelliptic curves over Q of genus n with a rational Weierstrass point in any 
large family are ordered by height, the average size of the 2-Selmer group is 3. 

Theorem 37 is proven via an appropriate sieve applied to the counts of G(Z)-orbits on V(Z) having 
bounded height which we obtained in Section 7. 

Note that the set of all hyperelliptic curves in (1) with indivisible coefficients is large. So too is the 
set of such hyperelliptic curves C : y 2 = f(x) whose Jacobians have semistable reduction. Any family 
of hyperelliptic curves in (1) defined by finitely many congruence conditions on the coefficients is also 
a large family. Thus Theorem 37 applies to quite general families of hyperelliptic curves. 

11.1 A weighted set U (F) in V(Z) corresponding to a large family F 

Proposition 13 implies that non-identity elements of the 2-Selmer group of the Jacobian of the hy- 
perelliptic curve C = C(c2, . . . , C2 n +i) over Q defined by (1) are in bijective correspondence with 
G(Q) -equivalence classes of irreducible locally soluble elements B 6 V(Z) having invariants 2 8 C2, 
2 12 c 3 , . . ., 2 4 ( 2n+1 )c 2n+ i; in this bijection, we have H (B) = 2 8n ^ 2n+1 ^H(C). In §7, we computed the 
asymptotic number of G(Z)-equivalence classes of irreducible elements B E V(Z) having bounded 
height. In order to use this to compute the number of irreducible locally soluble G(Q) -equivalence 
classes of elements B e V(Z) having invariants in 

{(2 8 c 2 , . . . , 2 A ^c 2n+1 ) : (c 2 , . . . , c 2n+1 ) E F inv } (44) 

and bounded height (where F is any large family), we need to count each G(Z) -orbit G(Z) ■ B with 
a weight of l/n(B), where n(B) is equal to the number of G(Z)-orbits inside the G(Q) -equivalence 
class of BinV(Z). 



41 



To count the number of irreducible locally soluble G(Z) -orbits having invariants in the set (44) and 
bounded height, where each orbit G(Z) ■ B is weighted by l/n(B), it suffices to count the number 
of such G(Z)-orbits of bounded height such that each orbit G(Z) • B is weighted instead by l/m(B), 
where 



here 0(f) denotes a set of orbit representatives for the action of G{Z) on the G(Q) -equivalence class 
of B in V{Z), and Aut Q (£') (resp. Aut z (5')) denotes the stabilizer of B' in G(Q) (resp. G(Z)). 
The reason it suffices to weight by l/m(B) instead of l/n(B) is that we have shown in the proof of 
Proposition 3 1 that all but a negligible number o(X^ 2n+3 ^^ 4n+2 ' ) ) of G(Z) -orbits having bounded height 
have trivial stabilizer in G(Q) (and thus also in G(Z)), while the number of hyperelliptic curves in F 
of bounded height is > x^ 2n+ ^^ An+2 \ 

We use U(F) to denote the weighted set of all locally soluble elements in V(Z) having invariants 
in the set (44), where each element of B £ U(F) is assigned a weight of l/m(B). Then we have 
concluded that the weighted number of irreducible G(Z)-orbits of height less than 2 8n ( 2n+1 )X in U(F) 
is asymptotically equal to the number of non-identity 2-Selmer elements of all hyperelliptic curves of 
height less than X in F. 

The global weights m(B) assigned to elements B £ U(F) are useful for the following reason. For 
a prime p and any element B £ V(Z P ), define the local weight m p (B) by 



where O p (B) denotes a set of orbit representatives for the action of G{Z P ) on the G(Q P ) -equivalence 
class of B in V(Z P ), and AvXq p (B) (resp. AvXi v (B)) denotes the stabilizer of B in G(Q P ) (resp. 
G(Z P )). Then by Proposition 22, we have the following identity: 



Thus the global weights of elements in U(F) are products of local weights, so we may express the 
global density of elements U(F) in V(Z) as products of local densities of the closures of the set U (F) 
in V(Z P ). We consider these local densities next. 

11.2 Local densities of the weighted sets U (F) 

Suppose that F is a large family of elliptic curves, and for each prime p, let F p denote the resulting 
family of curves defined by congruence conditions over Z p . Let U (F) denote the associated weighted 
set in V(Z), and let U P (F) denote the p-adic closure of U(F) in V(Z P ). We now determine the p- 
adic density of U P (F), where each element B £ U P (F) is weighted by l/m p (B), in terms of a local 
(p-adic) mass M P (V, F) involving all elements of Jac(C)/2Jac(C) for curves C in F over Q p . 

Proposition 38 Let J be the constant of Proposition 34, and let F be any large family of hyperelliptic 
curves. Then 





m 



(B) = l[m p (B). 



p 
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where 



ae 2Jac(C)(Q p ) 



Proof: Let S p (c 2 , ■ ■ ■ , c 2n+ i) C Vz p be a set consisting of one element in each soluble G(Z p )-orbit on 
the elements in V(Z P ) having invariants equal to 2 8 C2, . . . , 2^ 2n+1 ^c 2n+ i. Define the multiset U P (F) by 

U P (F) := |J Gf(Zp) ■ B. 

(c 2 ,...,C2n + l)eFf v 
BeSp(c2,..-,C2n+l) 



Now we have 

dB f dB 



Ib( 



(45) 



'b&j p {f) m p (B) J B& u P (F) m p (B)#Aut Zp (B) ' 

The last integral in (45) can be computed using a Jacobian change of variables; indeed, Proposition 34 
and the principle of permanence of identities imply that 

dB (46) 



B£Up(F) m p 



= |2^ 2 " +3 ^| p Vol(G(Z p )) f ,mA t (m d ° 2 ' ■ ■ dc2n+1 - 

For B G S p ( C2, • • • , c 2n+ \), let 5 = B , Bi, . . . , .Bfc be those elements in S p (c2, ■ ■ ■ , C2 n +i) that are 
p )-equivalent to 5. Let n p (B) denote the number of G(Z p )-orbits in the G(Q P ) -equivalence class 
of B in V(Z p ). Since m p {B) and n p (B) depend only the G(Q p )-class of 5, we have: 

1 1 1 1 /y. #Aut Qp (j?) V 1 ^ 



m p (S0#Aut Zp (5 i ) m p (B) ^ #Aut Zp ( J B i ) n p (B) V ^ #Aut Zj ,(5 i ) / ^ #Aut Zp (S i 



i=0 

A- 



n p (B i )#Aut Qp (B j ) 

For (c 2 , . . . , c 2n+ i) G Z 2n , Proposition 18 implies that if B is a soluble element in V(Q P ) having 
invariants c 2 , . . . , c 2 „+i, then there exists B' G V(Z p ) having invariants c 2 , . . . , c 2n+ i that is G(Q P )- 
equivalent to B. Thus, by Theorem 3, we see that ^ Be5p ( C2 C2 +1 ) - |s is equal to the cardinality of 
Jac(C(c2, . . . , C2n+l))(Qp )/2Jac(C(c2,... , C2n+i))(Qp)> and the cardinality of Aut.q (B) is equal to 
the cardinality of Jac(C(c 2 , . . . , c 2n +i))(Q P )[2]. Proposition 38 now follows. □ 

In the analogous manner, if F is a large family of hyperelliptic curves, then we may define M P (F) 
to be the measure of F p nv with respect to the measure dc 2 ■ ■ ■ dc 2n+ \ on Z p ™, where the measure dc m on 
Z p is normalized so that the total measure is 1. That is, we have 



M P (F)= / dc 2 ---dc 2n+1 . (47) 

Jc=C(c2,...,c 2n +i)&Fp 

In Section 9, we will be interested in comparing the masses M P (V, F) and M p (F). 
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11.3 A key uniformity estimate 

For each prime p, let W p denote the set of elements B in V(Z) such that p 2 | A(£>). Then Theorem 1 
of [2] implies: 

Proposition 39 For any M > 0, we have 

J2N(W P ;X) = 0(X/M), 

p>M 

where the implied constant is independent of X and M. 

11.4 Weighted count of elements in U(F) having bounded height 

For a large family F, we may now describe the asymptotic number of G(Z)-orbits in U(F) having 
bounded height. 

Theorem 40 Let F be any large family of hyperelliptic curves. Then N(U(F);2 8n ( 2n+1 ^X), the 
weighted number of G^L) -orbits in U (F) having height less than 2 8n ( 2n+1 )X, is given by 



n 2 m 



N(U(F); 2 M2n+l) X) = 2 4n(2n+3) . £ Vol(ftx(L ( ™^)) -pj f _J_ dv . X (2n+3)/(4n+2) 

m=0r=l 2m+n v Ju p (F) m p( V ) 



m=0 r=l p 



+ o(X (2n+3)/(4n+2) ). 

(48) 

Proof: By Theorem 24, Theorem 36, and the multiplicativity of weights (Proposition 22), it follows 
that for any fixed positive integer Y, we have 

N(V(Z) n [n p<Y u P (F)]-, 2 M2n.+i )x) _ " £; YoMn^L^)) n r 1 

\ I m =o t=1 p<Y JU P^> PK ' 

where V(Z) fl [n p< yf/ ; ,(F)] is viewed as a weighted set in which each element B is weighted by 
\/m{B). Letting Y tend to infinity, we obtain that 

TS P X(- + 3)/(4„ + 2) <^ 2m+ n UJ Up(F)mp (v) dV - ^ 

To obtain a lower bound for N(U(F); 2 8n ( 2n+1 ^X), we note that 

V(Z) n f) C/ P (F) C U |J IV P 



(even as weighted sets, since all weights in U P (F) are less than 1.) Hence, by the uniformity estimate 
of Proposition 39, we obtain 

N(U(F);2^X) y ~ Vol(tt.v(L^>)) . n f _J_ _ , 

m=0 t=1 p<y ' 
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t/p(.F) «M W J 



Letting Y tend to infinity completes the proof. □ 



It remains to evaluate expression (48) in terms of the total number of hyperelliptic curves in F 
having height less than X. 

12 Proof of Theorem 1 

12.1 The number of hyperelliptic curves of bounded height 

Theorem 41 Let F be a large family of hyperelliptic curves. Then the number of curves C in F with 
H(C) < X is given by 

[ dc 2 --- dc 2n+1 ■ 11 M p {F) ■ x( 2 " +3 )/( 4 -+ 2 ) + (x( 2 "+ 3 V( 4 "+ 2 )). 

J H(c 2 ,...,c 2n+1 )<X p 

This theorem is proven in much the same manner as Theorem 40 (but is easier), again using the unifor- 
mity estimate of Proposition 39. 

12.2 Evaluation of the average size of the 2-Selmer group 

We now have the following theorem, from which Theorem 37 (and thus Theorem 1) will be seen to 
follow. 

Theorem 42 Let F be a large family of hyperelliptic curves. Then we have 

v — "\ n 2*™ 

h (#S2< J> " l) 1 J| ' Vol(G=(Z)\G=M) -EE^r dc- dc M 

li m H( - C ^ <X _ m=0 T=1 ±A(o 2 ,...,c 2 „ +1 )>0 



> / dc 2 ■ ■ ■ do 

CGF t-^l J H(c 2 ,...,c 2n+1 )<X 

H(C)<X m=0 ±A(c 2 ,...,c 2 „ +1 )>0 



\J\ P ■ Vol( G -(Z p)) . / ^ TT-^T ^ ■ • ■ dc 2n+1 



>c=c( C2 ,.., C2n+1 )eF p j^ )(Qp) #MC)[2](Q P , 

n e 2 



/ dc 2 ■ 


■ ■ dc 2n+ i 


Jc=C(c2,...,C2„+i)€F p 





Proof: This follows by combining Proposition 38, Theorem 40 and expression (39) for the volume 
Vol(7e x (£ (m ' r) )), and Theorem 41. □ 

In order to evaluate the right hand side of the expression in Theorem 42, we use the following fact (see 
[14, Lemma 3.1]): 
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Lemma 43 Let J be an abelian variety over Q p of dimension n. Then 



#(J(Q P )/2J(Q P )) 



#J[2]{Qp) 
2"-#J[2](Q p ) 



ifp^2; 
ifp = 2. 



Proof: It follows from the theory of formal groups that there exists a subgroup M C J(Q P ) of finite 
index that is isomorphic to Z™ [34]. Let if denote the finite group J(Q p )/M . Then by applying the 
snake lemma to the following diagram 




we obtain the exact sequence 

-> M[2] -> J(Q P )[2] ->■ #[2] -> M/2M -> J(Q P )/2J(Q P ) -> if/2# ->■ 0. 

Since if is a finite group and M is isomorphic to Z" Lemma 43 follows. □ 

The expression on the right hand side in Theorem 42 thus reduces simply to the Tamagawa number 
r (G ss ) = Vol(G ss (Z)\G ss (M)) • n p Vol(G ss (Z p ) of G ss , which is 2. The completes the proof of 
Theorem 37, and thus of Theorem 1 . 

In fact, the proof of Theorem 1 shows more. For example, we may study the distribution of the 
nonidentity Selmer elements inside J(Q u )/2 J(Q V ) for any (finite or infinite) place v of Q. In this 
regard, we have: 

Theorem 44 Let F be a large family of hyperelliptic curves C of genus n whose associated invariants 
C2, • • • , C2n+i He in a v-adic disk small enough so that Jac(C)(Q I/ )/2Jac(C)(Q l/ ) is constant. Then 
when the hyperelliptic curves in F are ordered by height, the images of the non-identity 2-Selmer 
elements under the natural map 



S 2 (Jac(C)) Jac(C)(Q l/ )/2Jac(C , )(Q,) 



(50) 



are equidistributed. 



To prove Theorem 44, on the right side of the equation in Theorem 42, we replace the sum over all a in 
Jac(C) (Q„) / 2 Jac(C) (Q u ) by the corresponding sum over a lying in any subset S C Jac(C) (Q„) / 2 Jac(C) (Q„). 
This yields the average number of nonidentity Selmer elements that map to E, and we see that that the 
result is proportional to the size of S, proving Theorem 44. The same argument also allows one to show 
equidistribution of non-identity 2-Selmer elements in Ylues J ac (Q^)/2Jac(Q Jy ) for any finite set S of 
places of Q, provided that our large family F of hyperelliptic curves lies in the intersection of suffi- 
ciently small z/-adic discs (y G S) so that, for all C G F, the product Y\ ueS Jac(C)(Q i/ )/2Jac(C)(Q v ) 
is constant. 

There is an alternative interpretation of many of these calculations (and cancellations!) in terms of 
an adelic geometry-of-numbers argument. This interpretation was introduced by [38] and we briefly 
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describe it here. Let A be the ring of adeles of Q and let dv be the unique Haar measure on V(A) which 
is the counting measure on the discrete subgroup V(Q) and gives the compact quotient V(A)/V(Q) 
volume one. Let S be the vector group over Q of dimension 2n where the invariant polynomials take 
values, and let ds be the unique Haar measure on S(A) which is counting measure on the discrete 
subgroup S(Q) and gives the compact quotient S(A) / S(Q) volume one. 

Define the compact subset S(A) <X as the product of the elements in S(Z P ) that define a p- 
indivisible polynomial having non-zero discriminant with the elements in S(M) that define a poly- 
nomial having non-zero discriminant and bounded height H < X. Define the subset V(A) <X as the 
subset of elements in V(A) locally trivial everywhere and having characteristic polynomial in S(A) <X . 
The group G(Q) acts on V(A) <X and the orbit space has finite volume. The adelic volume formula is 
then 

vo\(V(A) <x /G(Q), dv) = 2- vol(S(A) <x , ds). 

The factor 2 in this formula is the Tamagawa nuumber of G: the volume of G(A)/G(Q) with respect 
to Tamagawa measure dg. The proof of this volume formula involves the compatibility of the three 
measures ds, dv, and dg (as in Proposition 34) as well as a product formula for the Herbrand quotients 
#(J(Q v )/2J(Qv))/#J['2](Qv) (as in Proposition 43). Geometry-of-numbers arguments (as in Sec- 
tions 8 and 9) and a sieve (as in Section 10) can then be used to estimate the average number (= 2) 
of irreducible locally soluble rational orbits, and so the average number of non-trivial classes in the 
2-Selmer group. 
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